Sussex's Professor Julia Simner is part of a €6.8m project to turn sounds into pictures for deaf people. One idea is a baby monitor that shows a baby's cry.
A University of Sussex scientist is part of a team that has won a €6.8 million European grant to turn everyday sounds into pictures that deaf people can understand at a glance, including a baby monitor that would let deaf parents see their baby crying.
Professor Julia Simner, professor of neuropsychology at the university’s Falmer campus, is one of three lead researchers on the project, called ViSONIC. It has been awarded a Synergy Grant by the European Research Council (ERC), shared with the French research body CNRS and Université Jean Monnet in Saint-Étienne.
The money runs for six years. The project will start no later than 1 April 2027, according to the announcement from Université Jean Monnet; the exact date is still being agreed with the ERC.
Sounds most people take for granted
Birdsong, a baby’s cry, music and an ambulance siren all tell a hearing person something about what is going on around them. The partners point out that these sounds remain largely out of reach for the more than 400 million people worldwide who live with disabling hearing loss.
ViSONIC aims to turn those environmental sounds into images that anyone can understand straight away, “without training”, in the words of the announcement.
Why a high note feels small
The work rests on links the brain makes between the senses without being taught. A high-pitched sound brings to mind a light colour, a small object or an upward movement, the partners say, and the project sets out to use those instincts rather than a code that people would have to learn.
That is Professor Simner’s field. The University of Sussex said she brings her experience in multisensory science to the team, building on the links our brains make between hearing and vision, “why a high-pitched sound feels ‘light’ or ‘small’, for example”.
Using acoustic analysis, artificial intelligence and perception experiments, the team will build a library of about 1,000 “sonic visuals”. It will then test how those pictures are understood by adults, children and babies; by hearing and deaf people; by synaesthetes, who experience one sense through another; and by people who are hypersensitive to sound.
According to the announcement, the results should show how the brain links vision and hearing, when that ability emerges as children grow, and whether it depends on having heard sound at all.
The three lead researchers are Professor Simner at Sussex, David Reby of the ENES bioacoustics lab at Université Jean Monnet in Saint-Étienne and Olivier Pascalis of CNRS in Grenoble, who works on developmental psychology.
A baby monitor you can see
The project is meant to produce working tools as well as research papers. The partners name three:
- a visual baby monitor that lets deaf parents “see” their baby’s cries;
- silent gaming interfaces for people who are distressed by certain sounds;
- a virtual-reality cultural event that brings deaf and hearing audiences together.
One of 48 out of 957
ERC Synergy Grants go to groups of two to four lead researchers who combine their skills on a single ambitious project. This year 48 projects were chosen from 957 applications, according to the announcement, which works out at about one bid in 20.
From dementia glasses to deaf parents
It is the second large prize this year for Professor Simner. In March she was lead scientist on CrossSense, AI smart glasses for people with early-stage dementia, which won the £1 million Longitude Prize on Dementia.
The glasses use augmented reality to label objects and flag hazards such as a hot hob, with a soft-spoken AI assistant called Wispy that guides people through tasks like making a cup of tea. When that prize was announced, the university reported that 250 people with lived experience of dementia had helped design the glasses.
Professor Simner said then: “It shows how technology works best when it’s guided by the people it’s meant to support.”
Have your say