IEEE Robotics & Automation Magazine - March 2016 - 96

Trial and Error

Using Previous
Experiences as
Simulation
Models
in Humanoid
Motor Learning

S

ince biological systems have the ability to efficiently
reuse previous experiences to change their behavioral
strategies to avoid enemies or find food, the number
of required samples from real environments to
improve behavioral policy is greatly reduced. Even
for real robotic systems, it is desirable to use only a limited
number of samples from real environments due to the limited
durability of real systems to reduce the required time to
improve control performance. In this article, we used previous
experiences as environmental local models so that the movement policy of a humanoid robot can be efficiently improved
with a limited number of samples from its real environment.
We applied our proposed learning method to a real humanoid
robot and successfully achieve two challenging control tasks.
We applied our proposed learning approach to acquire a policy for a cart-pole swing-up task in a real-virtual hybrid task
environment, where the robot waves a PlayStation (PS) Move
motion controller to move a cart-pole in a virtual simulator.
Furthermore, we applied our proposed method to a challenging basketball-shooting task in a real environment.

Digital Object Identifier 10.1109/MRA.2015.2511681
Date of publication: 23 February 2016

96

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

Image lIcensed by Ingram PublIshIng

By Norikazu Sugimoto,
Voot Tangkaratt,
Thijs Wensveen,
Tingting Zhao,
Masashi Sugiyama, and
Jun Morimoto

MARCh 2016

Motor Learning in Real Environments
For biological systems, exploring outside worlds for long periods of time can be dangerous, since this increases the probability of encountering enemies. On the other hand, a biological
system needs to explore its surrounding environment to sample the data and improve its behavioral policy to increase the
probability of survival. In such cases, efficiently reusing previous experiences is crucial for improving its behavioral policies
without actually interacting with real environments. A
standard approach is using a parameterized simulation model.
1070-9932/16©2016IEEE



Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2016

IEEE Robotics & Automation Magazine - March 2016 - Cover1
IEEE Robotics & Automation Magazine - March 2016 - Cover2
IEEE Robotics & Automation Magazine - March 2016 - 1
IEEE Robotics & Automation Magazine - March 2016 - 2
IEEE Robotics & Automation Magazine - March 2016 - 3
IEEE Robotics & Automation Magazine - March 2016 - 4
IEEE Robotics & Automation Magazine - March 2016 - 5
IEEE Robotics & Automation Magazine - March 2016 - 6
IEEE Robotics & Automation Magazine - March 2016 - 7
IEEE Robotics & Automation Magazine - March 2016 - 8
IEEE Robotics & Automation Magazine - March 2016 - 9
IEEE Robotics & Automation Magazine - March 2016 - 10
IEEE Robotics & Automation Magazine - March 2016 - 11
IEEE Robotics & Automation Magazine - March 2016 - 12
IEEE Robotics & Automation Magazine - March 2016 - 13
IEEE Robotics & Automation Magazine - March 2016 - 14
IEEE Robotics & Automation Magazine - March 2016 - 15
IEEE Robotics & Automation Magazine - March 2016 - 16
IEEE Robotics & Automation Magazine - March 2016 - 17
IEEE Robotics & Automation Magazine - March 2016 - 18
IEEE Robotics & Automation Magazine - March 2016 - 19
IEEE Robotics & Automation Magazine - March 2016 - 20
IEEE Robotics & Automation Magazine - March 2016 - 21
IEEE Robotics & Automation Magazine - March 2016 - 22
IEEE Robotics & Automation Magazine - March 2016 - 23
IEEE Robotics & Automation Magazine - March 2016 - 24
IEEE Robotics & Automation Magazine - March 2016 - 25
IEEE Robotics & Automation Magazine - March 2016 - 26
IEEE Robotics & Automation Magazine - March 2016 - 27
IEEE Robotics & Automation Magazine - March 2016 - 28
IEEE Robotics & Automation Magazine - March 2016 - 29
IEEE Robotics & Automation Magazine - March 2016 - 30
IEEE Robotics & Automation Magazine - March 2016 - 31
IEEE Robotics & Automation Magazine - March 2016 - 32
IEEE Robotics & Automation Magazine - March 2016 - 33
IEEE Robotics & Automation Magazine - March 2016 - 34
IEEE Robotics & Automation Magazine - March 2016 - 35
IEEE Robotics & Automation Magazine - March 2016 - 36
IEEE Robotics & Automation Magazine - March 2016 - 37
IEEE Robotics & Automation Magazine - March 2016 - 38
IEEE Robotics & Automation Magazine - March 2016 - 39
IEEE Robotics & Automation Magazine - March 2016 - 40
IEEE Robotics & Automation Magazine - March 2016 - 41
IEEE Robotics & Automation Magazine - March 2016 - 42
IEEE Robotics & Automation Magazine - March 2016 - 43
IEEE Robotics & Automation Magazine - March 2016 - 44
IEEE Robotics & Automation Magazine - March 2016 - 45
IEEE Robotics & Automation Magazine - March 2016 - 46
IEEE Robotics & Automation Magazine - March 2016 - 47
IEEE Robotics & Automation Magazine - March 2016 - 48
IEEE Robotics & Automation Magazine - March 2016 - 49
IEEE Robotics & Automation Magazine - March 2016 - 50
IEEE Robotics & Automation Magazine - March 2016 - 51
IEEE Robotics & Automation Magazine - March 2016 - 52
IEEE Robotics & Automation Magazine - March 2016 - 53
IEEE Robotics & Automation Magazine - March 2016 - 54
IEEE Robotics & Automation Magazine - March 2016 - 55
IEEE Robotics & Automation Magazine - March 2016 - 56
IEEE Robotics & Automation Magazine - March 2016 - 57
IEEE Robotics & Automation Magazine - March 2016 - 58
IEEE Robotics & Automation Magazine - March 2016 - 59
IEEE Robotics & Automation Magazine - March 2016 - 60
IEEE Robotics & Automation Magazine - March 2016 - 61
IEEE Robotics & Automation Magazine - March 2016 - 62
IEEE Robotics & Automation Magazine - March 2016 - 63
IEEE Robotics & Automation Magazine - March 2016 - 64
IEEE Robotics & Automation Magazine - March 2016 - 65
IEEE Robotics & Automation Magazine - March 2016 - 66
IEEE Robotics & Automation Magazine - March 2016 - 67
IEEE Robotics & Automation Magazine - March 2016 - 68
IEEE Robotics & Automation Magazine - March 2016 - 69
IEEE Robotics & Automation Magazine - March 2016 - 70
IEEE Robotics & Automation Magazine - March 2016 - 71
IEEE Robotics & Automation Magazine - March 2016 - 72
IEEE Robotics & Automation Magazine - March 2016 - 73
IEEE Robotics & Automation Magazine - March 2016 - 74
IEEE Robotics & Automation Magazine - March 2016 - 75
IEEE Robotics & Automation Magazine - March 2016 - 76
IEEE Robotics & Automation Magazine - March 2016 - 77
IEEE Robotics & Automation Magazine - March 2016 - 78
IEEE Robotics & Automation Magazine - March 2016 - 79
IEEE Robotics & Automation Magazine - March 2016 - 80
IEEE Robotics & Automation Magazine - March 2016 - 81
IEEE Robotics & Automation Magazine - March 2016 - 82
IEEE Robotics & Automation Magazine - March 2016 - 83
IEEE Robotics & Automation Magazine - March 2016 - 84
IEEE Robotics & Automation Magazine - March 2016 - 85
IEEE Robotics & Automation Magazine - March 2016 - 86
IEEE Robotics & Automation Magazine - March 2016 - 87
IEEE Robotics & Automation Magazine - March 2016 - 88
IEEE Robotics & Automation Magazine - March 2016 - 89
IEEE Robotics & Automation Magazine - March 2016 - 90
IEEE Robotics & Automation Magazine - March 2016 - 91
IEEE Robotics & Automation Magazine - March 2016 - 92
IEEE Robotics & Automation Magazine - March 2016 - 93
IEEE Robotics & Automation Magazine - March 2016 - 94
IEEE Robotics & Automation Magazine - March 2016 - 95
IEEE Robotics & Automation Magazine - March 2016 - 96
IEEE Robotics & Automation Magazine - March 2016 - 97
IEEE Robotics & Automation Magazine - March 2016 - 98
IEEE Robotics & Automation Magazine - March 2016 - 99
IEEE Robotics & Automation Magazine - March 2016 - 100
IEEE Robotics & Automation Magazine - March 2016 - 101
IEEE Robotics & Automation Magazine - March 2016 - 102
IEEE Robotics & Automation Magazine - March 2016 - 103
IEEE Robotics & Automation Magazine - March 2016 - 104
IEEE Robotics & Automation Magazine - March 2016 - 105
IEEE Robotics & Automation Magazine - March 2016 - 106
IEEE Robotics & Automation Magazine - March 2016 - 107
IEEE Robotics & Automation Magazine - March 2016 - 108
IEEE Robotics & Automation Magazine - March 2016 - 109
IEEE Robotics & Automation Magazine - March 2016 - 110
IEEE Robotics & Automation Magazine - March 2016 - 111
IEEE Robotics & Automation Magazine - March 2016 - 112
IEEE Robotics & Automation Magazine - March 2016 - 113
IEEE Robotics & Automation Magazine - March 2016 - 114
IEEE Robotics & Automation Magazine - March 2016 - 115
IEEE Robotics & Automation Magazine - March 2016 - 116
IEEE Robotics & Automation Magazine - March 2016 - 117
IEEE Robotics & Automation Magazine - March 2016 - 118
IEEE Robotics & Automation Magazine - March 2016 - 119
IEEE Robotics & Automation Magazine - March 2016 - 120
IEEE Robotics & Automation Magazine - March 2016 - Cover3
IEEE Robotics & Automation Magazine - March 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2010
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2010
https://www.nxtbookmedia.com