A Face to Remember

Researchers show that a tuning algorithm can make one’s profile photo more memorable.

By | April 17, 2014

MIT, KHOSLA ET AL.Standing out in a crowd—and particularly, the crowded world of social media—may become a little easier as MIT researchers develop a clever way to tweak profile photos to play up a person’s most memorable facial features. Aditya Khosla and others at MIT have found a way to make photographs of faces more memorable or more forgettable, opening new doors in the understanding of memory.

The process was detailed in work presented at the International Conference on Computer Vision in Sydney in December 2013 and earned Khosla a Facebook Graduate Fellowship for further research. According to the researchers, such “feature tuning”—which could be extended from memorability to other qualities, like confidence or trustworthiness—has the potential to change everything from one’s profile photo on Facebook to online dating to political campaigns and advertising strategies.

In Khosla’s first few weeks as a graduate student at MIT, the talk on campus was about bringing psychology together with machine learning—training a computer to identify patterns in data. Aude Oliva, a researcher with MIT’s computer vision and graphics group, explored what makes images memorable and came to an unlikely conclusion.

“They found that all images are actually not created equal,” says Khosla. “There are some that just stick in our mind: Memorability is an intrinsic property of an image.”

Oliva and her colleagues noticed that although people couldn’t guess which pictures would be memorable in an online test, certain images were consistently remembered better or worse. Their next line of research was to figure out what in a photo of a face makes it stick, or fade away. By inputting 2,222 photos and their memorability scores from the online test as training data, Oliva’s group was able to teach a computer to predict the memorability of photos it hadn’t seen before.

The photos that ranked highest on memorability didn’t have greater beauty, a bigger smile, a special gleam in the eye or a certain shape of the jaw. Rather, the computer was picking up on a multitude of tiny details and quirks that contributed to memorability but would be impossible to find and define on their own. There were no big distinct trends—instead, a thousand tiny tweaks, statistically tied to the memorability of the photos in the study.

 Machine learning often follows this process, although it hadn’t been used for memorability before, says Gregory Hager, a computer vision researcher at Johns Hopkins University.

“By and large, these sorts of approaches don’t give you explanatory power—they give you a result,” says Hager. “So you get an algorithm which might perform quite well, but you can’t articulate necessarily why it does that. It’s basically a black box that goes from measurements to output.” Knowing that something is measurable is the first step; after that, it’s possible to train a computer to look for it and put it to work.

Changing it up

Intrigued by the idea that small changes could lead to big differences in memorability, Khosla brought his idea to Oliva: If a photo is memorable because of differences in a few pixels, could changing those pixels change the memorability?

Khosla and his colleagues set out to create an algorithm to capture that memorability and impart it to other photographs. The researchers focused on faces because small changes could have a large effect while leaving the photo recognizable.

The team’s program first makes a chain of small, random changes to a photo, creating a slew of photos with only slight differences from the original. It gives each new image a score, detracting points if the changes affect attributes like emotion or attractiveness or if the face begins to look different from the original; it awards points if the photograph becomes more memorable. Meticulously measuring 10,000 randomly-changed versions, the program pinpoints the highest scorer: a more memorable face that still looks like the subject. Then it starts the process again and alters that picture, slowly building up to the best possible modifications. It can also go the other way — toward a less memorable photograph. As Khosla worked with the algorithm, he tried tuning other attributes besides memorability—the algorithm could just as easily tune things like attractiveness and friendliness, and it was easier to see at a glance if it was working.

Researchers tested the newly changed images on participants using an online memory game. Three out of four images they altered to be more or less memorable produced the desired reaction in real people.

The results are the first of their kind. Hager says that while the data collection process was similar to other machine learning work, combining that algorithm with an optimization process is new.

Branching out

Algorithm in hand, Khosla will soon turn his sights to tuning other, more complicated features of a photo using the same type of data gathered from online surveys. Then, the possibilities are endless, he says.

“Make me look more confident, for a dating website,” says Khosla. “Make me look more trustworthy, for a politician’s photo in a newspaper. There’s more than just the simple attributes that people have studied.”

Oliva says it’s important to remember that the measurements and alterations refer to the photograph, not the face. “It’s not necessarily that your face itself is memorable or forgettable,” she says, “it’s actually the attitude you have. The light coming from your face. Everyone, depending on how the photo was taken, can look more average or more distinctive.” She suggests that the process could be used to choose photos with better features, rather than just to alter them after the fact.

According to Adam Glenn, a journalist and digital media consultant, this kind of feature tuning—while technically impressive—fits with the way people currently use social media.

“I really see no difference between altering your image in this way and fudging your age on a dating site or polishing your resume to focus on your skills in a positive light,” he says. “It’s like virtual makeup, a way of presenting yourself.”

Glenn anticipates that the process could be very popular, especially when used with apps that rely on an image to draw you to a purchase or a romance.

Looking forward

In broader terms, the MIT researchers are already thinking about future uses. The benefits for advertising are clear to Khosla.

“If we could pick or modify images to make them extremely memorable, we’d just have to show them once,” he says, as opposed to over and over again on billboards, in magazines and on TV. “You don’t really have a choice but to store the information because that’s how we are all programmed.” Changing photos of products or packaging in smart ways could create an instant connection.

He’s quick to add that there are more meaningful uses for the memory research, as well. Researchers might develop a model tailored to help people who are unable to recognize faces, for instance, or educators could make more memorable charts and infographics for easier learning. Oliva’s group has already published an analysis of what makes infographics memorable and what parts of them people store in memory, as well as looking into the memorability of particular words.

Phillip Isola, a graduate student who works with Oliva on memorability, put his own image through the algorithm near the beginning of the project.

“If I were confident that it was really preserving my identity,” he says, “not modifying who I am but just making the best angle on me, then it’d be pretty nice to use for a profile pic.”

Of course, because it was an early version of the algorithm, it didn’t go perfectly: the computer had tried to insert teeth into his closed-mouth smile.

Khosla is confident that the technology will be useable by the mainstream soon—his next goal is to cut down on the time the algorithm takes and to reduce errors and artifacts introduced in the process. He says smoothing out the process only a matter of tinkering. Then and only then will it reach its final and inevitable form: the smartphone app.

This article is provided by Scienceline, a project of New York University's Science, Health and Environmental Reporting Program.
Advertisement

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: Donoby

Donoby

Posts: 1

April 17, 2014

One significant aspect of this is that users would need to "be careful what you wish for", as the algorithm only works on one facet or characteristic of the image. So for instance, increasing memorability may reduce apparent trustworthiness, increasing apparent confidence may decrease "good looks". Also, partly arising from the first point, it will not be possible to predict how people will react to the image. However the advance in machine learning is to be welcome, notwithstanding the fact that how/why it achieves it's result is not know. 

Avatar of: loverboy

loverboy

Posts: 2

April 17, 2014

Have you thought about the confounding factors that may have affected your data, for example maybe the population group you used to collect the data is from a certain cultural background/country/race thus making his/her memorability of the faces change, we obviously remember faces we haven't seen based on attractiveness, or some other similar feature which is probably controled by some part of the brain(eg- for attractiveness we probably use the limbic system), we remember them probably because they are the best mixture from a genetic point of view which is good to the human race as a whole from an evolutionary point of view, thus if you improve your data collection model, I feel maybe you could improve the precision of

Avatar of: loverboy

loverboy

Posts: 2

April 17, 2014

You could increase the accuracy based on the type of market for advertisement, eg- if you are advertising in specific region where certain type of people live takng into account everything that could effect the outcome, then bring out an average pattern memorable to all the different people.
Avatar of: Kirby

Kirby

Posts: 1

April 21, 2014

One quote “I really see no difference between altering your image in this way and fudging your age on a dating site or polishing your resume to focus on your skills in a positive light,” is just another way of lying for personal or professional gains. I make no ethical judgement about lying in this context. If lying to get a more positive opinion from others is ok, then fine, but let's call it what it is...lying. I am not sure if I were to interview a person for a position, if that person came in not looking like I expected, is the type I would hire. I would assume that this person believes that getting a foot in the door by lying is a valid professional choice. I would also assume that this person could apply this to other areas and do the same with data or outcomes.

Avatar of: blumberg

blumberg

Posts: 32

April 23, 2014

They basically took 10 pounds off of that Asian women and made her look more Western: thinned nose, thinned jaw-line, rounded eyes.  Not sure altering one's image to look more attractive is the best thing to do.  Pretty morally bankrupt if you ask me.

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement
RayBiotech
RayBiotech

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
ProteinSimple
ProteinSimple
Advertisement
NeuroScientistNews
NeuroScientistNews