A Face to Remember

MIT, KHOSLA ET AL.Standing out in a crowd—and particularly, the crowded world of social media—may become a little easier as MIT researchers develop a clever way to tweak profile photos to play up a person’s most memorable facial features. Aditya Khosla and others at MIT have found a way to make photographs of faces more memorable or more forgettable, opening new doors in the understanding of memory. The process was detailed in work presented at the International Conference on Computer Vision in Sydney in December 2013 and earned Khosla a Facebook Graduate Fellowship for further research. According to the researchers, such “feature tuning”—which could be extended from memorability to other qualities, like confidence or trustworthiness—has the potential to change everything from one’s profile photo on Facebook to online dating to political campaigns and advertising strategies. In Khosla’s first few weeks as a graduate student at MIT, the talk on campus was about bringing psychology together with machine learning—training a computer to identify patterns in data. Aude Oliva, a researcher with MIT’s computer vision and graphics group, explored what makes images memorable and came to an unlikely conclusion. “They found that all images are actually not created equal,” says Khosla. “There are some that just stick in our mind: Memorability is an intrinsic property of an image.” Oliva and her colleagues noticed that although people couldn’t guess which pictures would be memorable in an online test, certain images were consistently remembered better or worse. Their next line of research was to figure out what in a photo of a face makes it stick, or fade away. By inputting 2,222 photos and their memorability scores from the online test as training data, Oliva’s group was able to teach a computer to predict the memorability of photos it hadn’t seen before. The photos that ranked highest on memorability didn’t have greater beauty, a bigger smile, a special gleam in the eye or a certain shape of the jaw. Rather, the computer was picking up on a multitude of tiny details and quirks that contributed to memorability but would be impossible to find and define on their own. There were no big distinct trends—instead, a thousand tiny tweaks, statistically tied to the memorability of the photos in the study. Machine learning often follows this process, although it hadn’t been used for memorability before, says Gregory Hager, a computer vision researcher at Johns Hopkins University. “By and large, these sorts of approaches don’t give you explanatory power—they give you a result,” says Hager. “So you get an algorithm which might perform quite well, but you can’t articulate necessarily why it does that. It’s basically a black box that goes from measurements to output.” Knowing that something is measurable is the first step; after that, it’s possible to train a computer to look for it and put it to work. Changing it up Intrigued by the idea that small changes could lead to big differences in memorability, Khosla brought his idea to Oliva: If a photo is memorable because of differences in a few pixels, could changing those pixels change the memorability? Khosla and his colleagues set out to create an algorithm to capture that memorability and impart it to other photographs. The researchers focused on faces because small changes could have a large effect while leaving the photo recognizable. The team’s program first makes a chain of small, random changes to a photo, creating a slew of photos with only slight differences from the original. It gives each new image a score, detracting points if the changes affect attributes like emotion or attractiveness or if the face begins to look different from the original; it awards points if the photograph becomes more memorable. Meticulously measuring 10,000 randomly-changed versions, the program pinpoints the highest scorer: a more memorable face that still looks like the subject. Then it starts the process again and alters that picture, slowly building up to the best possible modifications. It can also go the other way — toward a less memorable photograph. As Khosla worked with the algorithm, he tried tuning other attributes besides memorability—the algorithm could just as easily tune things like attractiveness and friendliness, and it was easier to see at a glance if it was working. Researchers tested the newly changed images on participants using an online memory game. Three out of four images they altered to be more or less memorable produced the desired reaction in real people. The results are the first of their kind. Hager says that while the data collection process was similar to other machine learning work, combining that algorithm with an optimization process is new. Branching out Algorithm in hand, Khosla will soon turn his sights to tuning other, more complicated features of a photo using the same type of data gathered from online surveys. Then, the possibilities are endless, he says. “Make me look more confident, for a dating website,” says Khosla. “Make me look more trustworthy, for a politician’s photo in a newspaper. There’s more than just the simple attributes that people have studied.” Oliva says it’s important to remember that the measurements and alterations refer to the photograph, not the face. “It’s not necessarily that your face itself is memorable or forgettable,” she says, “it’s actually the attitude you have. The light coming from your face. Everyone, depending on how the photo was taken, can look more average or more distinctive.” She suggests that the process could be used to choose photos with better features, rather than just to alter them after the fact. According to Adam Glenn, a journalist and digital media consultant, this kind of feature tuning—while technically impressive—fits with the way people currently use social media. “I really see no difference between altering your image in this way and fudging your age on a dating site or polishing your resume to focus on your skills in a positive light,” he says. “It’s like virtual makeup, a way of presenting yourself.” Glenn anticipates that the process could be very popular, especially when used with apps that rely on an image to draw you to a purchase or a romance. Looking forward In broader terms, the MIT researchers are already thinking about future uses. The benefits for advertising are clear to Khosla. “If we could pick or modify images to make them extremely memorable, we’d just have to show them once,” he says, as opposed to over and over again on billboards, in magazines and on TV. “You don’t really have a choice but to store the information because that’s how we are all programmed.” Changing photos of products or packaging in smart ways could create an instant connection. He’s quick to add that there are more meaningful uses for the memory research, as well. Researchers might develop a model tailored to help people who are unable to recognize faces, for instance, or educators could make more memorable charts and infographics for easier learning. Oliva’s group has already published an analysis of what makes infographics memorable and what parts of them people store in memory, as well as looking into the memorability of particular words. Phillip Isola, a graduate student who works with Oliva on memorability, put his own image through the algorithm near the beginning of the project. “If I were confident that it was really preserving my identity,” he says, “not modifying who I am but just making the best angle on me, then it’d be pretty nice to use for a profile pic.” Of course, because it was an early version of the algorithm, it didn’t go perfectly: the computer had tried to insert teeth into his closed-mouth smile. Khosla is confident that the technology will be useable by the mainstream soon—his next goal is to cut down on the time the algorithm takes and to reduce errors and artifacts introduced in the process. He says smoothing out the process only a matter of tinkering. Then and only then will it reach its final and inevitable form: the smartphone app. This article is provided by Scienceline, a project of New York University's Science, Health and Environmental Reporting Program.

A Face to Remember

The Scientist ARCHIVES

Become a Member of

Related Topics

Meet the Author

Sarah Lewin

Related Research Resources

Why Do We Feel Butterflies in the Stomach?

Research Resources

Podcasts

Webinars

Videos

Infographics

eBooks

Long-Read RNA Sequencing Reveals a Regulatory Role for Splicing in Immunotherapy Responses

Research Roundtable: The Evolving World of Spatial Biology

Exploring the State of the Art in Gene Editing Techniques

Simplifying Early Alzheimer’s Disease Diagnosis with Blood Testing

Products

Product News

Research on rewiring neural circuit in fruit flies wins 2025 Eppendorf & Science Prize

EVIDENT's New FLUOVIEW FV5000 Redefines the Boundaries of Confocal and Multiphoton Imaging

EVIDENT Launches Sixth Annual Image of the Year Contest

10x Genomics Launches the Next Generation of Chromium Flex to Empower Scientists to Massively Scale Single Cell Research