Facebook bakes AI into its mobile app to power real-time artsy video filters

While Prisma might have dawned the artistic photo filter trend earlier this year, Facebook is now planning to make the same ubiquitous with the integration of AI into its mobile app. If you remember, we’d recently discovered that the social media giant was testing Prisma-like artsy filters via the native camera within its app.

Well, Facebook has today not only confirmed that it is working on new creative tools but also detailed the technology that’ll bring a new AI-powered camera feature to the official Facebook app. This technique called ‘style transfer’ will enable you to show off your creative side and convert your photos or videos into alluring Van Gogh(or Pablo Picasso, if you want) illustrations.

But, wouldn’t clicking, uploading, and converting(at off-shore data centers) the images/video consume an enormous amount of time?

Let me just say no and then go on to explain the most significant AI advancements that Facebook has been cooking behind the scenes. With the aim of putting state-of-the-art AI technology in the palm of your hands, the social media behemoth has developed a full-fledged deep learning network ‘Caffe2Go’ that has been embedded in the mobile app. Yes, the Facebook application. This updated app is currently not available to everyone but one should expect a global rollout very soon.

To add to your knowledge, Caffe2Go is a new deep learning platform that can capture, analyze and process pixels in real time on a mobile device. This is a particularly lightweight and modular industry-strength framework that ships at on four platforms with the same set of code: server CPU, GPU, iOS, and Android. The lean algorithm is not only modular but also scalable — implement on one, optimize for multiple.

Since Facebook is betting big on video, especially live video, it wanted to make the technology accessible in real-time to all of its users. Thus, the company has produced an amalgamation of two technologies: the Caffe2go runtime and style-transfer models to condense the size of the AI model ‘style transfer’ being baked in the app. This reduction in the size of this AI model allows the deep neural networks to process your videos in real-time, i.e 100x faster, on both Android and iOS.

Talking about the technology in its official blog post, Facebook engineers Yangqing Jia and Peter Vajda say,

Because our AI teams deal with both algorithms and large-scale systems, they were well suited to develop new models for both pursuits, making the style-transfer experience high-quality and fast.

It took both technologies to make it possible for you to feel like you have Van Gogh’s paintbrush in your hand when you pick up your phone to shoot a video.

This is a cutting edge advancement in artificial intelligence technology and the underlying deep neural networks that are synonymous to the brain cells. Over the past three months, the research team at Facebook has not only managed to develop an exceptional deep neural network but also embed the same into a mobile app on your smartphones.

Thus, Facebook is making complete use of the computational power of the neural networks to expand the scope of creative tools available to users on the platform. The company has created gesture-based controls which allow you the computer where you’re pointing and respond by activating different styles(or commands.) It will also be able to recognize your facial expressions to apply a certain artistic filter and click a selfie for you.

With Caffe2Go, AI has opened the door to new ways for people to express themselves,

says Facebook CTO Mike Schroepfer.

Machine learning is an important aspect of the 10-year roadmap Facebook detailed at the F8 developer conference earlier this year. It now adds that Caffe2Go is core to the machine learning products at Facebook and it is planning to integrate and roll out the same across the complete Facebook stack.

We are also committed to sharing our software and designs with the community so we can learn to better utilize the characteristics of multiple hardware platforms and algorithm designs, which is particularly important in cross-platform machine learning systems. We will be looking to open source parts of this AI framework over the coming months,

reads the blogpost.