IEEE Consumer Electronics Magazine - March 2018 - 93

We considered two categories of sound parameters: temporal and spectral. We used AM concerning the temporal parameters, which could be applied on a continuous sound and gave
rhythmic information in terms of the number of modulations.
Concerning spectral parameters, we considered brightness and pitch. Brightness is highly correlated to the spectral centroid [22]. Interestingly, Houben [19] suggested that
the spectral centroid may be linked to the perception of the
velocity of wooden balls. Pitch has also been widely studied
for sonification and is one of the fundamental sound attributes [23].

SONIFICATION STRATEGIES
We based all of our strategies on pink noise, as it is particularly suited to human perception. The two dynamic parameters
associated with the VO (velocity and VO-to-hand distance)
were sonified by the three sound parameters (pitch, brightness, and AM). We obtained the pitch by using a bandpass filter, varying between 300 and 900 Hz. We added an exponential
compression before the filtering, because we hypothesized
logarithmic pitch perception, as Parseihian et al. [8] and
Denjean et al. [9].
Brightness was composed of a combination of a highpass filter varying between 300 and 900 Hz, and a low-pass
filter. We created the AM parameter by multiplying the noise
by a low-frequency signal with a range between 1 and 10 Hz.
A precise description can be found on this article's companion page (https://www.prism.cnrs.fr/AdditionalContent/
VirtualObjectSonification/). We generated the sounds using
the Max MSP environment developed by Cycling '74 (https://
cycling74.com/).
As described previously, we highlighted interesting
parameters that could be used for sonification: two dynamic
parameters and three sound parameters. These five can be
combined to create the six simple associations, or sound
strategies, presented in Table 1. In this table, the VO-to-hand
distance and the VO velocity are associated with sound
parameters to create different sonification strategies. For
instance, strategy 1 corresponds to the sonification of handto-VO distance by the pitch strategy, with no sonification of
the VO velocity.

EXPERIMENT 1: MAPPING SELECTION
An essential question remaining was the manner by which to
link dynamics and sound parameters. The aim of this first
experiment was to find the type of mapping that was best
understood or preferred with respect to the sonified parameter so we could build an intuitive sonification strategy.

MAPPING DESIGN
In this study, we chose to test three different mapping functions, i.e., logarithmic, linear, and exponential, with two
polarities, i.e., positive and negative, leading to six different
mappings represented in Figure 1. The VO dynamics and
sound parameters were normalized between 0 and 1.

PROTOCOL
During the experiment, participants could see a screen that
included both their virtualized hand and action on the VO. A
spatialized sound feedback was given via three Fostex PM0.4d
loudspeakers to create a more natural immersion in the scene.
The loudspeakers were positioned in front of the participants
and on their left, and the spatialization was created using Spat
software (http://forumnet.ircam.fr/fr/produit/spat/).
For each strategy, the six curves in Figure 1 were tested in
a paired comparison task. This led to (6 · 5) /2 = 15 comparisons for each strategy and then to 15 · 6 sound strategies = 90
comparisons for each subject. Each pair consisted of comparing two mappings for a given sonification strategy. Within
each trial, a first mapping was presented to the subjects, who
were asked to perform three or four right-handed swipes with
their palm, with different amplitudes so they could hear all of
the sound variations linked to the sonified dynamic parameter. The mapping was then changed, and the subjects were
again asked to perform three or four swipes. Afterward, the
experimenter asked the subjects which of the two sounds
seemed the most natural in terms of matching between the
visual and auditory information.
To prevent a presentation-order bias, we divided the 20
subjects into two groups of ten, who compared the pairs in a
reversed order (from B to A instead of from A to B). All
paired comparisons were tested by the subjects in a randomized order.

STATISTICAL PROCESSING
To interpret the results, we chose the Bradley-Terry test,
which is particularly suited for data comparison. This test
gives a normalized preference score for each variable compared to the others.

RESULTS
The results of this experiment are presented in Figure 2. Each
sound strategy is represented by a row, and each mapping is

Table 1. The coding of the six sound strategies.
Sound Strategy

Distance

Velocity

Pitch

PARTICIPANTS

Brightness

Twenty volunteers-six women and 14 men, ages 21-44
(mean age, 28.7; standard deviation, 7.0), participated in this
experiment. They all worked for the Groupe PSA and reported no hearing problems.

Pitch

Brightness

METHOD

march 2018

IEEE Consumer Electronics Magazine

http://forumnet.ircam.fr/fr/produit/spat/ https://www.prism.cnrs.fr/AdditionalContent/VirtualObjectSonification https://www.prism.cnrs.fr/AdditionalContent/VirtualObjectSonification https://cycling74.com/ https://cycling74.com/

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - March 2018