Book Excerpt from Physical Intelligence

In Chapter 2, “Surfaces,” author Scott Grafton describes the ancient process of action perception, which helps organisms navigate their environments.

Jan 13, 2020
Scott Grafton
Pantheon, January 2020

Off-trail rambling is primordial, a physical skill we have used for as long as we have been a species, and showing it off can still be immensely pleasing.

To move through the world like this as one desires, the mind has to recognize what is possible or impossible. It takes some effort to grasp how acutely the brain tracks the surfaces of the physical world, the literal hard constraints that limit what is possible. And there are loads of these surfaces. Imagine taking a walk through a majestic California redwood forest. What do you see? You might describe a few of the trees, the noise of the birds, unusual ferns, or a cloud in the sky. These verbal descriptions tend to revolve around the objects of the forest. Now close your eyes and continue your stroll. As you take a few more steps, other features of the world take hold: the exact positions of all the trees near your path, the slope of the trail, obstructive rocks, looming branches, and the sheen on the ground from raindrops that make the trail slippery. These features are totally absent from the conscious narrative of the world you were creating just a minute earlier. Some features, like slipperiness, aren’t even objects. Most of these new elements are built from surfaces, textures, and edges that define action boundaries. Together, they set very hard limits on what you will be able to do. You can’t walk through trees, your foot needs to be elevated over the rocks in the path, you’ll want to avoid stepping on ice, and your head needs to duck below a limb. These surfaces are essential features that anyone must sense if they want to move through any landscape.

Eyes open, you can run down the trail and, with little awareness or effort, readily maneuver through an endless array of surfaces requiring specific actions—for instance, hopping atop rocks or lunging through deep snow. The surfaces constantly require on-the-fly adjustments so your feet end up in the right spots. This unconscious side of sensing, the one that takes in features in the world to allow you to move through it or on it, constitutes a fundamental kind of “action perception.” The philosopher Jakob von Uexküll was one of the first to recognize the close connection between what is perceived and what is possible when, in 1909, he coined the term “Umwelt.” He realized that space itself is weighted with meaning that profoundly shapes an organism’s behavior. Both a person and a squirrel will track spatial distances and textures as they navigate through the world, but they construct a profoundly different sense of meaning from those features, leading to a very different understanding of what their physical environment is.

In the 1970s J. J. Gibson was one of the first psychologists to point out that our environment is rich with action-relevant properties, readily detected by our sensory systems. On a walk through the redwoods, a tree trunk across the trail is as solid and impenetrable as a brick wall. There is no decision-making about whether it is possible to walk through a tree or a brick wall, and human vision rarely makes mistakes about classifying a redwood as a barrier to action. The kind of perception that is used for action operates on many spatial scales. Slopes are sensed, cliff edges are avoided, objects are deemed graspable or not. A hiker will respond to a small rock in the trail up ahead as something that needs to be stepped over. No conscious mental gymnastics are needed to adjust walking cadence, to place, for instance, the lead foot at exactly the right distance from the rock. It just happens; the rock is sensed and refines the ongoing gait. Gibson was fascinated by this direct, unconscious process. It is profoundly different from the conscious experience and verbal descriptions of the world that we recognize through introspection and observation. Gibson’s ideas emerged at a time when psychologists treated much behavior as a set of increasingly complicated reflexes, with animals just learning to make stereotypical and predictable responses to specific stimuli. Gibson tipped the logic of this by arguing that behavior is not just triggered by stimuli, it is constrained by stimuli. Seeing a redwood tree out in front doesn’t make a person do a specific behavior. It prevents her from doing many alternate behaviors. Similarly, the gaps between trees all form opportunities for possible paths to take. In this light, almost everything in the physical environment provides relevant information for possible as well as impossible behavior.

Action perception is a deep, fundamental aspect of all organisms that are capable of movement, from the most complex vertebrates to some of the simplest of invertebrates. Take, for example, action perception in jellyfish. They typically have different types of eyes that likely serve different purposes, like helping them flee from a shadow that might be a predator or letting them move toward light. But that kind of stimulus-response behavior only requires the sensing of light, and there is nothing to suggest the creature has knowledge about the external world, a representation of surfaces or shapes that could influence its behavior. And then, there is the box jellyfish, Tripedalia cystophora. Unlike jellyfish living near sandy shores or open ocean, the box jellyfish lives in a complex maze of mangrove swamps. And unlike the others, the box jellyfish possess an impressive array of twenty-four eyes. These eyes are special as well because they have spherical lenses like those of a fish and sensors on the back of the eye, like a camera, all hooked up to a tangle of neurons forming an extremely rudimentary nervous system. To see what the jellyfish could do, Anders Garm in Lund, Sweden, made a swimming box for his jellyfish that had a steady current running through it. When he put dark rods downstream in the water, the jellyfish handily avoided them by swimming upstream. But when Garm used clear rods, the box jellyfish bumped into the obstacles and pinballed through the gaps, proving they had used sight to evade the dark rods. By varying the width of the dark rods, he could measure the jellyfishes’ visual acuity, roughly a finger width. He also tested a similar species, Chiropsella bronzie, which lives in a simpler natural environment consisting of a sandy bottom and far fewer underwater obstacles. Poor Chiropsella: it managed to avoid some of the visible black rods but with much less finesse than the mangrove-dwelling Tripedalia, repeatedly bashing into even the fattest obstacles. The difference between the two species of jellyfish reveals how direct sensing can drastically expand an organism’s ability to act in an environment by converting what is sensed into a representation of what is possible and impossible. Whereas crude visual detection is sufficient for Chiropsella, whose few natural obstacles are easily spotted from a distance, the Tripedalia has transcended a life of simple stimulus-driven reflexes into one with elemental action perception, allowing it to move in a far more complex world.

Excerpted with permission from Physical Intelligence: The Science of How the Body and the Mind Guide Each Other Through Life by Scott Grafton (Pantheon, January 2020).