I am currently investigating a little-known form of synaesthesia, whereby visual motion or flashes evoke faint auditory sensations. I have a personal interest in this, because I experience this phenomenon myself.
By now, most people have seen the ‘jumping pylons’ gif, which evokes juddering sounds in many people. This sound is probably driven by the expectation of something heavy hitting the ground, in combination with the camera shake effect.
Credit: HappyToast
However in people with strong ‘visual ears’, even abstract motion or flashing can evoke sounds. For example, I ‘hear’ car indicator lights, flashing shop displays, animated adverts on web-browsers, lip-movements, and the footsteps of people as they walk. It is a clear auditory sensation, mostly in my mind’s ear, though sometimes I can confuse it with real sounds if the latter are very quiet. The sounds are like white noise (‘sshhh’), but often they have different harmonics, especially when there are sequences of flashes. For example alternating flashing beacons on either side of a zebra crossing might sound like ‘shiii’, ‘shuuu’. It’s interesting but it can get pretty distracting, especially at the end of a tiring day. I have now met quite a few people who experience this too, so perhaps I’m not completely bonkers!
The demos below simulate (very crudely!) the kinds of sounds that might be experienced. I have added sounds, generated by measuring the motion energy from the video frames (how much light is changing over space and time) and using that to modulate white noise. The result only resembles very crudely what I typically experience, but enough to give an idea.
My research so far has made three main discoveries about this phenomenon, which I call the visually-Evoked Auditory Response (vEAR, or visual-ear synaesthesia):
- Illusory sounds evoked by motion can interfere with detection of real sounds (Fassnidge et al, 2017)
- vEAR seems to have high prevalence: up to 20% (Fassnidge & Freeman, 2018)
- Visual and auditory brain areas may cooperate, rather than compete, in people with visual ears (Fassnidge et al, 2019).