Rainbows in my ears—Synesthetic color perception with partial-reduced and morphed musical instrument timbres

Background: Synesthesia is a perceptual phenomenon in which sensory inputs
automatically elicit consistent additional sensations. Sound-color synesthesia,
one of the most widely recognized forms, involves experiencing colors and
shapes in response to auditory stimuli. Within this, timbre-color synesthesia,
reported in 26% of music-color synesthetes, has rarely been studied in
depth, particularly regarding how specific timbre manipulations influence
color perception.
Methods: This study investigated whether changes in timbre-induced color
perception occur continuously or abruptly as sound parameters are altered,
and whether generalizable patterns can be identified. For the first time, E
color similarity calculations were applied to objectively assess participants’
timbre-color synesthesia. Participants listened to loudness-matched single
tones (flute, oboe, French horn, violin, and piano) from the Vienna Symphonic
Library, presented both in their original form and with manipulated timbres.
Manipulations included partial reduction (removing partials from the first to the
tenth) and morphing between instrument pairs across seven stages (from 100:0%
to 0:100%). After each sound, participants described their color perception using
a color selection field, generating RGB and HSL values of the chosen colors.
Results: Findings revealed correlations between the individual color perceptions.
Increasing partial reduction was associated with decreased saturation and
increased lightness, particularly for flute, oboe, violin, and piano. Using audio
features such as spectral centroid, harmonicity strength, and percussive
loudness, generalizable rules for the perception of timbre-induced colors in
timbre-color synesthetes could be observed. For morphed sounds, perceived
colors shifted progressively from those associated with the initial instrument
toward those associated with the target instrument across the morphing stages.
Here, too, spectral centroid, harmonicity strength, and percussive loudness
showed strong overall correlations between instrumental sounds and the colors
they induce.
Conclusion: This study demonstrates that manipulating musical instrument
timbre through partial reduction and morphing induce clear but individual
patterns of color perception in timbre-color synesthetes, which differ
significantly from the responses of non-synesthetes.