IEEE Systems, Man and Cybernetics Magazine - April 2023 - 30

with densely labeled tracks that allows more fine-grained
and audiovisual analysis for AD.
Dataset
Overview of the MuMMER Dataset
The MuMMER dataset [35] was built to facilitate re -
search in robot perception, particularly for the evaluation
of perception approaches. It consists of audiovisual
recordings of people interacting with the robot in an
open environment using a WoZ approach. The data are
collected using the Pepper robot [48]. As shown in Figure
2(a), the robot has one frontal color camera and one
Asus Xtion depth camera that works at 8 (respectively, 5)
frames per second (fps) at a resolution of 640 × 480. In
addition, during the recording, an external moving Intel
D435 camera was placed on the top of the tablet, and a
static Kinect v2 camera was fixed behind the robot to
shoot the entire scene. Figure 2(b)-(d) shows the view of
the three camera settings. With both static and moving
cameras, the dataset has the advantage of both: the Intel
one moves as the robot is moving, while the Kinect one
is static. The Kinect camera and D435 were set to 15 fps
at resolutions of 960 × 540 and 1,280 × 720, respectively.
The audio streams are recorded with the four-channel
microphone arrays, with the Kinect and the robot at a
frame rate of 48 kHz. The technical details can be found
in the MuMMER article [35].
The MuMMER dataset scenario was designed such that
two or more participants interact with the robot (chatting,
quizzing, performing a guidance task, telling the news, or
telling jokes), while additional people are simply spectating.
Unlike most HRI datasets, the MuMMER dataset
allows the interactors to speak to each other and the
robot. These scenarios involve a multiparty conversation
Kinect
Intel
R
(a)
(b)
between the robot and people. Different participants were
involved in the conversation to make the scenario more
realistic. These participants are passersby in the background
troublemakers trying to distract the robot's attention,
people leaving the field of view when the robot is
giving a direction, and, potentially, people coming back
after a while to tell the robot about the early direction or
recommendation it made earlier. These settings make the
MuMMER dataset unique and more realistic. The dataset
has been recorded, split into 33 sessions, for a total duration
of 1 h 29 min. The short session is 1 min 6 s, and the
longest session spans up to 5 min 6 s. Each frame consists
of a maximum of nine people. During the scenario recording,
28 participants and 22 protagonists took part. Most
frames have two people, while 25% of them have at least
one spectator not interacting with the robot [35]. The
MuMMER dataset consists of 80,488 Kinect color frames,
80,865 Kinect depth frames, 47,023 robot color frames,
23,450 robot depth frames, 80,346 Intel color frames, and
80,310 Intel depth frames. Out of these, it consists of
506,713 annotated faces with their identities.
E-MuMMER Dataset
Ignoring the depth frames, we only selected the Kinect
color frames and Intel color frames, for a total of 2 h 58 min.
We skipped the video recorded by the robot color camera
for two reasons: it was recorded at 8 fps, which does not
match with the other cameras, and it consists of fewer
frames that contain face regions than the Kinect and
Intel cameras. Most of the frames miss the speaking person's
face because humans stand very close to the robot
[shown in Figure 2(b)], and their upper body is not visible.
DaVinci Resolve software [49] has been utilized to resolve
audio and visual synchronization problems during the scenario
recording.
Addressee Annotation
The addressee labels are generated by
human annotators using the interface
shown in Figure 3. Each frame consists of a
bounding box around visible faces. The
activity timeline below the window depicts
the audio waveform to label speech start
and end timestamps. The speech segment
start and end timestamps ([(, ),tt
(, ), ,tt f (, )])tt
se
se
11
sn en
00
were selected manually.
The activity of each face was labeled
according to the activities within that
boundary, where ts0
timestamp of the segment at index zero, teo
(c)
(d)
Figure 2. (a) An example scene and camera settings. (b)-(d) The
views from the three cameras [35]: the (b) robot (R), (c) Kinect V2,
and (d) Intel D435.
30 IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE April 2023
indicates the end timestamp segment at
index zero, and n is the size of the speech
segment in the video.
There are two speech activity types in
E-MuMMER: " Speaking to the robot/
addressing the robot, " labeled as " 0, " and
indicates the start
IEEE Systems, Man and Cybernetics Magazine - April 2023

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - April 2023
