ABOVE: Researchers studied the brains of mice that had lost a fight and uncovered a key role for oxytocin in regulating social avoidance learning. © iStock, Georgejason
Like humans, mice live in complex social groups in which learning to distinguish between who to approach and who to avoid is key. Scientists knew that after losing a fight, defeated mice quickly learn to shy away from their victorious opponents, but the neural mechanisms that drive these behavioral changes were not fully understood.1
Recently, researchers led by Dayu Lin, a neuroscientist at New York University, characterized a hypothalamic brain circuit that regulates social avoidance in mice and uncovered the essential role of oxytocin in facilitating neural plasticity after a single defeat.2 These findings, published in Nature, support the idea that the brain’s oxytocin modulates more than just prosocial behaviors in animals.
“They discovered a very different set of oxytocin neurons and oxytocin-receptor-expressing neurons that have never really been studied before,” said Brian Trainor, a behavioral neuroscientist at the University of California, Davis, who was not involved in the study.
When Lin started this research almost five years ago, she did not plan to study the brain circuits mediating social avoidance. “It was actually a project related to maternal aggression,” she explained. At the time, Lin wanted to investigate whether oxytocin released in the ventrolateral part of the ventromedial hypothalamus (VMHvl), a brain region implicated in aggressive behavior and social fear, mediated the increase in female aggression during lactation, a critical behavioral change that enables a mother to protect her offspring from threats.3-5
After data from a year of experiments negated that hypothesis, the researchers explored new ideas. While brainstorming with her team, Lin remembered an experiment she completed as a postdoctoral researcher a few years earlier. One day, out of curiosity, Lin looked at the expression of c-fos, a marker of neuronal activity, in the VMHvl of animals that had lost a fight. The c-fos profile in those brains resembled the oxytocin receptor patterns in the VMHvl of mouse mothers. This similarity intrigued Lin, who hypothesized that these oxytocin-receptor-expressing (OXTR) neurons could have a role in defeat-induced behaviors.
To test this idea, Lin and her team exposed male and female mice to an aggressive same-sex conspecific for 10 minutes and assessed the animals’ behavior and brain activity before and after they lost a fight. The researchers focused on the anterior part of the VMHvl (aVMHvl), as previous findings from Lin’s team indicated that cells in this region become active in mice defending themselves against aggressors.6
Fiber photometry recording of the OXTR neurons in the aVMHvl revealed increased cell responses when the researchers exposed the defeated animals to their bullies confined inside a perforated cup. ”Defeated animals spend most of [their] time away from the aggressor, in comparison to before the defeat, when they spend most of the time close to the aggressor,” Lin explained.
To assess the functional relevance of these cells to defeat-induced behaviors, the team optogenetically activated aVMHvl OXTR neurons in mice that had never experienced a fight. Stimulation of these cells in naïve animals induced strong shunning from a restrained animal, suggesting that increased activity of these neurons drives social avoidance. In contrast, optogenetic inhibition of these cells in defeated animals increased the time they spent investigating their confined bullies, indicating that these neurons are necessary to induce avoidance in mice that lost a fight.
Since these cells expressed oxytocin receptors, the team next sought the source of oxytocin for the aVMHvl neurons. By mapping the expression of c-fos and oxytocin after defeat, the researchers found that oxytocin-producing cells from different areas of the hypothalamus were engaged, with the highest percentage of them belonging to a region called the retrochiasmatic supraoptic nucleus (SOR).
Additional experiments looking at the SOR oxytocin-producing cells revealed that these neurons fired when the mice lost a fight, but not when the defeated animals later encountered their bullies. These cells increased their activity when the animals experienced different painful stimuli, such as being poked or pinched with a tweezer, suggesting that SOR oxytocin-producing cells convey sensory information about discomforting aspects of the fight. “I've never seen anyone publish anything about those cells,” Trainor said. “They really discovered something that was sitting there in plain sight.”
Electrophysiological recordings in brain slices further showed that stimulation of SOR oxytocin-producing cells led to membrane depolarization of aVMHvl neurons. Addition of an oxytocin receptor agonist to the brain slices increased the likelihood of aVMHvl cells firing in response to an excitatory input, suggesting that the neuropeptide strengthens synaptic transmission within the aVMHvl OXTR cells.
Moving forward, Lin plans to continue studying this circuit to uncover other neuronal populations and brain regions that may contribute to social avoidance learning. She also wants to examine this brain circuit in more naturalistic settings, such as during the establishment of hierarchy within a group of mice.
According to Trainor, the study findings reinforce the idea that oxytocin’s role in the brain goes beyond modulating prosocial behaviors, and that the neuropeptide acts as an enhancer of social experiences, making good experiences better and bad experiences worse. “It fills in this gap of oxytocin having these social avoidance types of effects,” Trainor said. “A lot of people have a hard time grasping that. Hopefully, this work will change some minds in the field.”
- Qi CC, et al. Interaction of basolateral amygdala, ventral hippocampus and medial prefrontal cortex regulates the consolidation and extinction of social fear. Behav Brain Funct. 2018;14(1):7.
- Osakada T, et al. A dedicated hypothalamic oxytocin circuit controls aversive social learning. Nature. 2024;626(7998):347-356.
- Lin D, et al. Functional identification of an aggression locus in the mouse hypothalamus. Nature. 2011;470(7333):221-226.
- Silva BA, et al. Independent hypothalamic circuits for social and predator fear. Nat Neurosci. 2013;16(12):1731-1733.
- Georgescu T, et al. Prolactin-mediated restraint of maternal aggression in lactation. Proc Natl Acad Sci U S A. 2022;119(6):e2116972119.
- Wang L, et al. Hypothalamic control of conspecific self-defense. Cell Rep. 2019;26(7):1747-1758.e5.