IEEE Robotics & Automation Magazine - December 2013 - 110

We also evaluate how the amount of time in advance
given to the users relates to the quality of synchronization.
Figure 7 shows the results
of this analysis. The chart
corresponds to an experiWe explore anticipation for ment where each user is
invited to play in sync
human-robot interaction
with a strike performed
in three different ways: 1)
in the context of robotic
only downward movement, 2) upward and
musicianship.
downward movements,
and 3) gesture plus
upward and downward
movements (the gesture consisted of a left-right movement).
The results show that the synchronization does not
improve considerably when including a gesture previous to
the upward movement, but being able to see the upward
movement before the strike is very important.
It is worth mentioning that all time measurements for the
experiments reported were performed by comparing the
responses of two different piezo sensors that detect the
moment of strike. Therefore, the delay related to the computer
vision apparatus is included. The observation that a strike is
about to happen involves detecting an upward or downward
movement, which is negligible in computational terms. Likewise, the computations that lead to the parameters describing
the time of strike and the drum to be hit are negligible.
There were no particular criteria for the selection of
human subjects involved in the study. The fact that most had
musical experience was not a problem in our opinion, in fact,
it might enhance the importance of the results. However,
because the experiment is very simple and requires no musical training, we think that musicians generally perform better
than nonmusicians. However, this question deserves more
experimental analysis.
Conclusions and Future Work
In this article, we explored anticipation for robotic musicianship. Anticipation was based on computer vision. Infrared
LEDs were attached to the ends of drum mallets, and an
infrared camera (actually the Wii Remote) was used to track
them. The strike of the mallets was anticipated using uniformly accelerated trajectory prediction.
Two different algorithms for anticipation were described,
obtaining predictions of ~10-100 ms before the strike event.
We also showed in a user test that, on average, synchronization was performed better by the system algorithm robot
than by humans, and that, although anticipation was important for human players, the time interval used for it can be
relatively short.
Our reported experiments and results were very encouraging. Although technically simple, the presented implementation already highlighted the potential of anticipation in
human-robot music ensembles. We find that the application
of visual-cues-based anticipation algorithms is of great impor110

IEEE ROBOTICS & AUTOMATION MAGAZINE

DECEMBER 2013

tance in human-robot interaction, particularly in the realm of
musical performance. This technique can also be artistically
explored for composing innovative music pieces.
An interesting avenue for future work is the study of
visual-based anticipation in contexts with other musical
instruments (for instance, in the case when a guitarist changes
chords) and/or with other musical robots. In addition, other
algorithms, such as gesture recognizing algorithms, can be
used for trajectory prediction and human-robot interaction
in mallet-based robots (like Shimon).
Acknowledgments
We would like to thank Lory Wilson for providing the woodworking tools for the Wii Remote tripod, the mallets, LED
attachments, and the drum pedestals. We would also like to
thank the National Science Foundation for providing financial support.
References
[1] B. Johansson and C. Balkenius, An Experimental Study of Anticipation in
Simple Robot Navigation (Lecture Notes in Computer Science). Berlin:
Springer-Verlag, 2007, vol. 4520, pp. 365-378.
[2] F. Eyssel, D. Kuchenbrandt, and S. Bobinger, "Effects of anticipated humanrobot interaction and predictability of robot behavior on perceptions of
anthropomorphism," in Proc. 6th Int. Conf. Human-Robot Interaction, ACM,
New York, 2011, pp. 61-68.
[3] M. Gielniak and A. Thomaz, "Anticipation in robot motion," in Proc.
IEEE Int. Symposium Robot Human Interactive Communication, 2011, pp.
449-454.
[4] G. Hoffman. (2010). Anticipation in human-robot interaction. presented at
AAAI Spring Symp. Series [Online]. Available: http://www.aaai.org/ocs/index.
php/SSS/SSS10/paper/view/1192
[5] H. Hoffmann, "Perception through visiomotor anticipation in a mobile
robot," Neural Netw., vol. 20, no. 1, pp. 22-33, Jan. 2007.
[6] Z. Wang, C. Lampert, K. Mulling, B. Scholkopf, and J. Petters, "Learning
anticipation policies for robot table tennis," in Proc. IEEE/RSJ Int. Conf. Intelligent Robots Systems, 2011, pp. 332-337.
[7] G. Weinberg and S. Driscoll, "Toward robotic musicianship," Comput.
Music J., vol. 30, pp. 28-45, Dec. 2006.
[8] G. Weinberg, A. Raman, and T. Mallikarjuna, "Interactive jamming with
Shimon: A social robotic musician," in Proc. 4th ACM/IEEE Int. Conf. Human
Robot Interaction, New York, 2009, pp. 233-234.
[9] G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the
OpenCV Library, 1st ed. O'Reilly Media: Sebastopol, CA, Oct. 1, 2008.
[10] M. Puckette, T. Apel, and D. Zicarelli, "Real-time audio analysis tools for Pd
and MSP," in Proc. Int.Computer Music Conf., San Francisco, 1998, pp. 109-112.

Marcelo Cicconet, Center for Music Technology, Georgia Institute of Technology, Atlanta. E-mail: cicconet@gmail.com.
Mason Bretan, Center for Music Technology, Georgia Institute
of Technology, Atlanta. E-mail: masonbretan@gmail.com.
Gil Weinberg, Center for Music Technology, Georgia Institute of Technology, Atlanta. E-mail: gilw@gatech.edu.

http://www.aaai.org/ocs/index

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2013