Twitter is Now Using Machine Learning to Auto Crop Images for the Best Highlights

Twitter machine learning auto image cropping sounds like crazy sci-fi stuff, but it actually produces some great results which are (hopefully) pleasing to the eye…

We often think about machine learning in the context of a robot beating a human at a game of chess. Or, contextual word processing for automated replies. But, those big applications often produce less than desirable results. In fact, machine learning delivers best when it comes to small, subtle user experiences. And, that seems to rightly be the case with Twitter’s use of neural networks to automatically crop photos.

Twitter Machine Learning Auto Image Cropping Rolls Out

Twitter announced this tool on its official blog. The microblog states that million of images are uploaded to the network every single day. What’s more, these images come in different shapes and sizes. Twitter states this very phenomenon presents a real challenge in delivering a consistent UI or user interface experience.

ML researcher Lucas Theis and ML lead Zehan Wang explain the company previously relied face detection to pull into focus the most prominent faces in images. But, this technique presents a problem because not all photos contain faces. What’s more, the old technology sometimes missed faces or identified faces in images where no actual faces appear.

The solution to this set of problems is what Twitter calls “cropping using saliency.” (Saliency means the most important thing within a photo.) Using academic studies about eye-tracking, to identify what people look at first, the new system trains on the most interesting aspect. With this data, Twitter uses machine learning to identify what’s most important in a given image. “This lets us perform saliency detection on all images as soon as they are uploaded and crop them in real-time.”

The new technology is currently rolling out to the desktop platform, as well as on iOS and Android. See more photos of the Twitter machine learning auto image cropping in action here.

Owen E. Richason IV

Covers social media, apps, search and like news. History buff, movie and theme park lover. Blessed dad and husband. Owen is also a musician and is the founder of Groove Modes.