IEEE Robotics & Automation Magazine - March 2016 - 103
time at regular intervals. The length of one trial was 0.5 s
with 0.002-s time steps [13].
The task setup is shown in Figure 9. The position of the goal
was 2 m from the robot. When the ball (weight: 0.5 kg, radius:
0.11 m) crossed a horizontal plane at height of 1 m, a terminal
reward is given based on the horizontal distance between the
ball and the basket:
U=
a 0 exp 6- a 1 d
@,
2
(11)
where the distance is defined as d 2 = p x2 + ^ p y - 2h2. The
ball's positions on the horizontal plane are p x and p y .
a 0 (= 100)and a 1 (= 5) are constant parameters. The control
cost was given as follows:
7
2
c (x (t), u (t)) = b / (i j (t) - i des
j (t)) ,
j=1
(12)
where constant parameter b was 5 # 10 - 4 . The state-dependent cost was given only in the terminal condition.
For one data set, the numbers of trials, databases, and recursive updates were N = 10, L = 10, and S = 10, respectively. The policy was updated with discount factor
c = 0.999 and learning rate f = 0.05.
The learning performance is shown in Figure 10. The learning converged around the 80th iteration. After the learning
stage, the successful shooting rate was 100%; 50 of 50 shots
went in.
The means and standard deviations of all the joint trajectories of the humanoid right arm in the basketball-shooting task
are shown in Figure 11. This result shows that properly coordinated joint movements had to be learned for a successful basketball-shooting task. On the other hand, small variations of
the movement trajectories were observed due to the uncertainty of the real system. The acquired behavior after the learning is shown in Figure 12.
Figure 12. The acquired behavior of the humanoid robot in the
basketball-shooting task. (Photos courtesy of ATR.)
Discussion
In our PGPE-based approach, the policy's output is deterministic and does not need noisy control output for exploration.
Thus, the control output
can be smooth, and
We applied our proposed
smooth control output is
highly suitable for robot
method to a challenging
hardware. In comparison,
the previously proposed
basketball-shooting task in
model-based policy update algorithms must first
a real environment.
identify the real environment [1], [4], [14], [17],
[22], [35], [36]. Therefore,
exploratory noisy input using a real robot is mandatory, although successful applications of model-based RL methods to
real systems have been presented [14], [18]. In addition, the
previous approaches used an approximated dynamics model
with a function approximator, such as Gaussian process
march 2016
*
IEEE ROBOTICS & AUTOMATION MAGAZINE
*
103
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2016
IEEE Robotics & Automation Magazine - March 2016 - Cover1
IEEE Robotics & Automation Magazine - March 2016 - Cover2
IEEE Robotics & Automation Magazine - March 2016 - 1
IEEE Robotics & Automation Magazine - March 2016 - 2
IEEE Robotics & Automation Magazine - March 2016 - 3
IEEE Robotics & Automation Magazine - March 2016 - 4
IEEE Robotics & Automation Magazine - March 2016 - 5
IEEE Robotics & Automation Magazine - March 2016 - 6
IEEE Robotics & Automation Magazine - March 2016 - 7
IEEE Robotics & Automation Magazine - March 2016 - 8
IEEE Robotics & Automation Magazine - March 2016 - 9
IEEE Robotics & Automation Magazine - March 2016 - 10
IEEE Robotics & Automation Magazine - March 2016 - 11
IEEE Robotics & Automation Magazine - March 2016 - 12
IEEE Robotics & Automation Magazine - March 2016 - 13
IEEE Robotics & Automation Magazine - March 2016 - 14
IEEE Robotics & Automation Magazine - March 2016 - 15
IEEE Robotics & Automation Magazine - March 2016 - 16
IEEE Robotics & Automation Magazine - March 2016 - 17
IEEE Robotics & Automation Magazine - March 2016 - 18
IEEE Robotics & Automation Magazine - March 2016 - 19
IEEE Robotics & Automation Magazine - March 2016 - 20
IEEE Robotics & Automation Magazine - March 2016 - 21
IEEE Robotics & Automation Magazine - March 2016 - 22
IEEE Robotics & Automation Magazine - March 2016 - 23
IEEE Robotics & Automation Magazine - March 2016 - 24
IEEE Robotics & Automation Magazine - March 2016 - 25
IEEE Robotics & Automation Magazine - March 2016 - 26
IEEE Robotics & Automation Magazine - March 2016 - 27
IEEE Robotics & Automation Magazine - March 2016 - 28
IEEE Robotics & Automation Magazine - March 2016 - 29
IEEE Robotics & Automation Magazine - March 2016 - 30
IEEE Robotics & Automation Magazine - March 2016 - 31
IEEE Robotics & Automation Magazine - March 2016 - 32
IEEE Robotics & Automation Magazine - March 2016 - 33
IEEE Robotics & Automation Magazine - March 2016 - 34
IEEE Robotics & Automation Magazine - March 2016 - 35
IEEE Robotics & Automation Magazine - March 2016 - 36
IEEE Robotics & Automation Magazine - March 2016 - 37
IEEE Robotics & Automation Magazine - March 2016 - 38
IEEE Robotics & Automation Magazine - March 2016 - 39
IEEE Robotics & Automation Magazine - March 2016 - 40
IEEE Robotics & Automation Magazine - March 2016 - 41
IEEE Robotics & Automation Magazine - March 2016 - 42
IEEE Robotics & Automation Magazine - March 2016 - 43
IEEE Robotics & Automation Magazine - March 2016 - 44
IEEE Robotics & Automation Magazine - March 2016 - 45
IEEE Robotics & Automation Magazine - March 2016 - 46
IEEE Robotics & Automation Magazine - March 2016 - 47
IEEE Robotics & Automation Magazine - March 2016 - 48
IEEE Robotics & Automation Magazine - March 2016 - 49
IEEE Robotics & Automation Magazine - March 2016 - 50
IEEE Robotics & Automation Magazine - March 2016 - 51
IEEE Robotics & Automation Magazine - March 2016 - 52
IEEE Robotics & Automation Magazine - March 2016 - 53
IEEE Robotics & Automation Magazine - March 2016 - 54
IEEE Robotics & Automation Magazine - March 2016 - 55
IEEE Robotics & Automation Magazine - March 2016 - 56
IEEE Robotics & Automation Magazine - March 2016 - 57
IEEE Robotics & Automation Magazine - March 2016 - 58
IEEE Robotics & Automation Magazine - March 2016 - 59
IEEE Robotics & Automation Magazine - March 2016 - 60
IEEE Robotics & Automation Magazine - March 2016 - 61
IEEE Robotics & Automation Magazine - March 2016 - 62
IEEE Robotics & Automation Magazine - March 2016 - 63
IEEE Robotics & Automation Magazine - March 2016 - 64
IEEE Robotics & Automation Magazine - March 2016 - 65
IEEE Robotics & Automation Magazine - March 2016 - 66
IEEE Robotics & Automation Magazine - March 2016 - 67
IEEE Robotics & Automation Magazine - March 2016 - 68
IEEE Robotics & Automation Magazine - March 2016 - 69
IEEE Robotics & Automation Magazine - March 2016 - 70
IEEE Robotics & Automation Magazine - March 2016 - 71
IEEE Robotics & Automation Magazine - March 2016 - 72
IEEE Robotics & Automation Magazine - March 2016 - 73
IEEE Robotics & Automation Magazine - March 2016 - 74
IEEE Robotics & Automation Magazine - March 2016 - 75
IEEE Robotics & Automation Magazine - March 2016 - 76
IEEE Robotics & Automation Magazine - March 2016 - 77
IEEE Robotics & Automation Magazine - March 2016 - 78
IEEE Robotics & Automation Magazine - March 2016 - 79
IEEE Robotics & Automation Magazine - March 2016 - 80
IEEE Robotics & Automation Magazine - March 2016 - 81
IEEE Robotics & Automation Magazine - March 2016 - 82
IEEE Robotics & Automation Magazine - March 2016 - 83
IEEE Robotics & Automation Magazine - March 2016 - 84
IEEE Robotics & Automation Magazine - March 2016 - 85
IEEE Robotics & Automation Magazine - March 2016 - 86
IEEE Robotics & Automation Magazine - March 2016 - 87
IEEE Robotics & Automation Magazine - March 2016 - 88
IEEE Robotics & Automation Magazine - March 2016 - 89
IEEE Robotics & Automation Magazine - March 2016 - 90
IEEE Robotics & Automation Magazine - March 2016 - 91
IEEE Robotics & Automation Magazine - March 2016 - 92
IEEE Robotics & Automation Magazine - March 2016 - 93
IEEE Robotics & Automation Magazine - March 2016 - 94
IEEE Robotics & Automation Magazine - March 2016 - 95
IEEE Robotics & Automation Magazine - March 2016 - 96
IEEE Robotics & Automation Magazine - March 2016 - 97
IEEE Robotics & Automation Magazine - March 2016 - 98
IEEE Robotics & Automation Magazine - March 2016 - 99
IEEE Robotics & Automation Magazine - March 2016 - 100
IEEE Robotics & Automation Magazine - March 2016 - 101
IEEE Robotics & Automation Magazine - March 2016 - 102
IEEE Robotics & Automation Magazine - March 2016 - 103
IEEE Robotics & Automation Magazine - March 2016 - 104
IEEE Robotics & Automation Magazine - March 2016 - 105
IEEE Robotics & Automation Magazine - March 2016 - 106
IEEE Robotics & Automation Magazine - March 2016 - 107
IEEE Robotics & Automation Magazine - March 2016 - 108
IEEE Robotics & Automation Magazine - March 2016 - 109
IEEE Robotics & Automation Magazine - March 2016 - 110
IEEE Robotics & Automation Magazine - March 2016 - 111
IEEE Robotics & Automation Magazine - March 2016 - 112
IEEE Robotics & Automation Magazine - March 2016 - 113
IEEE Robotics & Automation Magazine - March 2016 - 114
IEEE Robotics & Automation Magazine - March 2016 - 115
IEEE Robotics & Automation Magazine - March 2016 - 116
IEEE Robotics & Automation Magazine - March 2016 - 117
IEEE Robotics & Automation Magazine - March 2016 - 118
IEEE Robotics & Automation Magazine - March 2016 - 119
IEEE Robotics & Automation Magazine - March 2016 - 120
IEEE Robotics & Automation Magazine - March 2016 - Cover3
IEEE Robotics & Automation Magazine - March 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2010
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2010
https://www.nxtbookmedia.com