IEEE Consumer Electronics Magazine - March 2018 - 92

vision, which is not ideal while driving, since it can distract
and endanger the driver.
To solve this problem, we reimagined the manipulation of
the central interface system. In this article, we investigate a
virtual HMI controlled by gestures. Gestures can be performed blind and while driving. To replace the visual feedback, we propose a sonification process that would provide
information about the nature of the manipulated function or
the degree of interaction, such as a change of temperature of
1, 2, or 3 °C.
Sonification, defined as "the use of nonspeech audio to
convey information or perceptual data" [1], could be a guide
to reach a particular gesture. Sonification based on artificial
sounds has been used for localization to help blind people
cross the street or even follow more complex paths [2]. It has
also been instituted for guidance tasks in medicine, surgery
[3], and rehabilitation [4]-[7]. A formal comparison of sound
strategies in a guidance undertaking was conducted by Parseihian [8], who evaluated the efficiency of several sound
parameters in terms of rapidity, precision, and overshooting
the target. Closer to the automotive context, acoustic parking
aids are now integrated in most cars to give the driver feedback on the distance to an obstacle. Sonification in electric
vehicles, where engine noise is absent, is also relevant for
informing the driver about the car's acceleration [9]. The use
of gestures in conjunction with sonification for HMI manipulation has significant potential [10]. For example, Jakus et al.
have shown that auditory interfaces can be comparable to
visual ones for everyday manipulations and are beneficial for
driving performance and perceived workload [11].
Even though virtual reality (VR) was invented many
years ago, it is only recently that technology improvements
have allowed it to become more integrated in various systems [12]. VR can open up new perspectives, facilitating
complete immersion in a virtual world [13], and lead to
genuine appreciation of assorted systems [14]. In our case,
VR is of interest for HMI, allowing the introduction of a
VO that can be manipulated with gestures. Thanks to the
VO, it is not the gestures that are sonified but their consequences. The VO is the sound source, and it can be considered an intermediate between the user's gesture and the
HMI. This is of importance for the naturalness of the sonification procedure, since direct gesture sonification is found
highly unnatural from an ecological point of view [15],
[16]. Gestures usually do not produce sound, but their consequences do-e.g., an object moving along a surface. In
our system, the VO's dynamic information is transmitted
through sound, which highlights the importance of which
sonification strategy to choose.

THE VO
A central aspect of our study is the manipulation of a VO.
The VO is justified from an ecological point of view, i.e., in
everyday life, sounds are more frequently created by interactions between objects rather than by the objects themselves.
Thus, it is natural to sonify a gesture's consequence rather
92 IEEE Consumer Electronics Magazine

march 2018

than the gesture itself. In this study, in which we chose the
swipe gesture to highlight the VO sonification, the VO is
linked to the hand by a spring with a force vf = - k $ vx a, where
k is the stiffness of the spring, a is arbitrarily set by the
author, and x is the hand-to-VO distance. In our experiments,
the subjects performed their swipes with their right hand in
one direction, from right to left. Consequently, during a
swipe movement, the VO was pushed by the subject's hand
and then continued its trajectory because of its inertia. As the
VO-to-hand distance increased, the force of the spring progressively stopped the VO movement and automatically sent
it back to the subject's hand. It is thereby possible to interact
with the VO without seeing it.

SONIFICATION STRATEGIES
In our case, we wanted the sound to provide information about
the movement of the VO with respect to the user's gesture. We
call these parameters related to VO movement dynamic or VO
parameters. The sonification strategies based on amplitude
modulation (AM), pitch, and brightness are then associated
with each of the VO parameters described below.

VO PARAMETERS
From a physical point of view, VO movement can be modeled by the following equation, with m the VO mass and k
the stiffness of the spring: xp + ~ 20 · x a = 0, where ~ 0 = k/m .
This equation shows that the dynamics of the VO are influenced by the VO mass and spring stiffness, which will be
the parameters of interest to inform the users in the sonification process.
Hence, the choice of VO parameters should be consistent
with the perception of the VO movement. From a perceptual
point of view, velocity seems to be a good parameter for sonification. For instance, Thoret et al. [18] sonified the velocity
of a pen on a graphical tablet to recreate the perception of a
natural friction sound. Houben's work [19]-[21] showed that
the perception of the velocity of wooden balls was conveyed
by a spectral parameter. Moreover, the physically based sonification strategies pertaining to a rolling object proposed by
Rath et al. [17] highlighted the fundamental role of velocity
in the perception of VO dynamics.
Second, as the VO was linked to the hand by a spring,
we investigated sonification of the VO's relative position
with respect to the hand. Furthermore, as the task of experiment 2 was based on judgment of the VO-to-hand distance,
it made sense to evaluate a sonification strategy implying
this parameter.

SOUND PARAMETERS
Our idea was to use a continuous sound linked to the VO to
represent its behavior and provide information about its current state. Consequently, we designed sonification strategies
that could be applied to a stationary sound. In the next experiments, we chose to use a pink noise as the initial stationary
sound so we could make general conclusions and not influence our subjects with the sound's nature or esthetics.

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - March 2018