Advertisement
Gene Tools
Gene Tools

The Sound of Color

A completely colorblind musician and painter perceives the world in a new way with help from technology.

By | May 1, 2012

image: The Sound of Color ORANGE TONES: Neil Harbisson wearing his “eyeborg,” a device that converts light waves into vibrations that lend a touch of color to his worldDan Wilton/RedBulletin

ORANGE TONES: Neil Harbisson wearing his “eyeborg,” a device that converts light waves into vibrations that lend a touch of color to his world DAN WILTON / REDBULLETIN

As a kid growing up in Barcelona, Spain, Neil Harbisson could tell you that the sky was blue, the grass was green, and a lemon was yellow. But he couldn’t tell you exactly what all those descriptions really meant. Born with a rare inherited condition  similar to the one that plagued the Pacific islanders neurologist Oliver Sacks wrote about in The Island of the Colorblind, Harbisson sees only in shades of gray, and had simply memorized the colors he thought he was supposed to know. But it wasn’t until he was 11 years old that he learned that he didn’t perceive the world in the same way as most people.

“I noticed that other students at school could identify colors easier than me,” he recalls. “Then I knew there was a problem with color.”

A decade later, as a music composition student at the Dartington College of Arts in England, Harbisson discovered there was hope that he might see things differently. In 2003, he attended a lecture about using technology to change the way we see the world. After the talk, Harbisson approached the speaker, young cybernetics innovator Adam Montandon, then at the University of Plymouth, to describe his condition and ask if there might be a way to help him perceive color.

On his train ride home, Montandon thought about the possibility of using a system that would assign different musical instruments to colors—drums for red and violins for blue, for example. But realizing that this would introduce his own interpretations of color, Montandon then considered a device that would simply say the names of colors aloud, but this didn’t sit right with him either. “I wanted to give him something a bit more magical,” Montandon recalls. Finally, he thought about the physical similarities of light and sound. “Light is a wavelength that moves very fast,” he says. “[If] you slow it down enough, it stops becoming visible. It starts becoming audible.”

In just 2 weeks’ time, Montandon and Harbisson created a device that translated the light waves that correspond to different colors into sounds with different pitches. The prototype, constructed from an inexpensive computer webcam, a laptop carried in a backpack, and a pair of old headphones, was a bit “crude,” Montandon admits. “It was fairly primitive, but it was good enough,” he says.

EYES AND EARS: The original version of the eyeborg used a computer webcam to detect light, a laptop to compute the conversion to sound, and headphones to transmit that sound to Harbisson’s ears.
EYES AND EARS: The original version of the eyeborg used a computer webcam to detect light, a laptop to compute the conversion to sound, and headphones to transmit that sound to Harbisson’s ears.
AN WILTON / REDBULLETIN

Harbisson tested the device in a school hallway in front of a big red notice board as a friend pointed to different objects, identifying their colors. He quickly learned the color/pitch associations. “Then he just ran off down the corridor,” Montandon says. “I couldn’t stop him. He went to listen to absolutely everything.” And when Montandon returned to the school 2 weeks later, he learned that Harbisson “hadn’t even switched off the computer.”

Harbisson listened to the colors of the houses in the street. He went to local grocers’ shops to listen to the sounds of fruits and vegetables. “It’s like listening to electronic music,” says Harbisson, who now wears a refined version of the device, which he calls an “eyeborg.”

“And it’s constant,” he adds. “I wasn’t expecting that I would be listening to colors all the time, but there’s color absolutely everywhere.”

The device has since been through many iterations, with the help of Montandon; a software developer from Kranj, Slovenia, named Peter Kese; and most recently, Matias Lizana, a computer engineering student at the Universitat Politècnica de Catalunya in Barcelona. Today, the system comprises a camera that sticks out above Harbisson’s head like an antenna, and a small computer chip that converts light to sound. Harbisson has since also forgone the need for headphones, mounting the chip to the back of his head, where it transmits the sound vibrations directly into his skull bones. “I receive color through the bone, and I’m listening to you through the ears,” Harbisson says. This helps him “differentiate what is a visual sound and what is an audio sound.”

Harbisson can also hear many more colors than the handful portrayed by the original device. He can even hear colors of wavelengths not visible to the human eye, such as those in the infrared. This year, he plans to add ultraviolet. The project is the ultimate demonstration of the promise of cyborgs, says Montandon, now a digital technologies professor at Erhvervsakademiet Lillebælt in Denmark. “It’s very easy with the technology we have now to explore different parts of the spectrum that we don’t normally experience,” Montandon adds. “You’re not just enabling someone with a disability, but you’re enabling someone to be more than a regular human.”

Of course, the device is not the most practical medical solution to color blindness, its creators recognize. First, wearing a camera around all day is inconvenient. Furthermore, the device is really designed for those with true color blindness, or achromatopsia, which affects only about 1 in 30,000 people—not for those who simply are unable to distinguish red from green, a far more common condition. And people with achromatopsia “also have a whole host of other vision problems,” says Medical College of Wisconsin neuroscientist Joseph Carroll, including involuntary eye movements called nystagmus, very poor acuity, and photophobia. “The ability to perceive color is the least of their worries,” he says. A more promising treatment for these patients is gene therapy to restore cone function, and thus increase acuity and reduce photophobia, he adds. Human trials for these treatments are expected to start this year.

That said, Carroll adds, “this is really cool and adds a dimensionality to your visual experience.” Indeed, Harbisson’s eyeborg is catching the attention of some musicians and artists. The pianist Jools Holland, for example, used a version of the device to accompany live concerts held in 2009–2010 across the U.K., Montandon says. “He likes to improvise, [so] we created a reverse system that would turn his music into colors and lights as he played.”

And Harbisson himself, who now works full-time as a freelance “colorologist,” pairing colors with music, uses the device for artistic inspiration, often painting what he hears. Listening to his world has definitely changed how he perceives it, he says. “People said that cities were gray—they’re not. They are actually extremely colorful. I’m discovering color in a different way.”

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: rcmcfd

rcmcfd

Posts: 1

May 3, 2012

Interesting - artificial synesthesia. 

Avatar of:

Posts: 0

May 3, 2012

Interesting - artificial synesthesia. 

Avatar of:

Posts: 0

May 25, 2012

rcmcfd - I had almost the same thought: Do the sounds and colors of this device map onto the experiences of people with synesthesia? 

Avatar of:

Posts: 0

May 25, 2012

Hmm, that's a lot of bandwidth for a rather trivial application. Peter Meyer started his interface for blind persons (The vOICe) more than 20 years ago if I am right. Google "seeing with sound" and you'll find a youtube video...
The vOICe is sensory substitution, i.e., the visual cortex can be "reprogrammed" to employ more auditive information (some blind can hear walls and windows because of reflections), but this takes a long training period. Here the only interesting question may be whether only the color "circuitry" in the visual cortex can be reprogrammed.
Hans du Buf

Avatar of: Ken Pimple

Ken Pimple

Posts: 10

May 25, 2012

rcmcfd - I had almost the same thought: Do the sounds and colors of this device map onto the experiences of people with synesthesia? 

Avatar of: HansDuBuf

HansDuBuf

Posts: 1

May 25, 2012

Hmm, that's a lot of bandwidth for a rather trivial application. Peter Meyer started his interface for blind persons (The vOICe) more than 20 years ago if I am right. Google "seeing with sound" and you'll find a youtube video...
The vOICe is sensory substitution, i.e., the visual cortex can be "reprogrammed" to employ more auditive information (some blind can hear walls and windows because of reflections), but this takes a long training period. Here the only interesting question may be whether only the color "circuitry" in the visual cortex can be reprogrammed.
Hans du Buf

Avatar of: Ichha Purak

Ichha Purak

Posts: 1457

May 26, 2012

A person who has not actually visualized colours in real sense can imagine and give his own vision and format of any colour as when he touches ripe tomatoes he imagines it is of red colour and so on because some body has narrated this fact to him

May 26, 2012

 Am not sure synesthesia experiences are shared across individuals.  For people who give numbers color, for instance, it's individual, although there may be similarities among affected family members.

Avatar of:

Posts: 0

May 26, 2012

A person who has not actually visualized colours in real sense can imagine and give his own vision and format of any colour as when he touches ripe tomatoes he imagines it is of red colour and so on because some body has narrated this fact to him

Avatar of:

Posts: 0

May 26, 2012

 Am not sure synesthesia experiences are shared across individuals.  For people who give numbers color, for instance, it's individual, although there may be similarities among affected family members.

Avatar of:

Posts: 0

June 1, 2012

Impressive discovery and invention, blind people can benefit from it.

Avatar of: Technispace

Technispace

Posts: 1

June 1, 2012

Impressive discovery and invention, blind people can benefit from it.

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
Rainin Instrument
Rainin Instrument
Advertisement
PITTCON
PITTCON
Life Technologies