IEEE Robotics & Automation Magazine - March 2016 - 98

PGPE Policy Updates
In our work, we use PGPE-based policy updates [6], [33]. Policy parameter w is stochastically sampled from prior distribution p (w | t) with hyperparameter t. In other words, the
policy is deterministic, but its parameter is stochastic.
In PGPE, hyperparameter t is optimized to maximize expected return J (t) (Table 1, 1.1). Optimal hyperparameter
)
t) is given by t := argmax t J (t) . In practice, a gradient
method is used to find t ): t # t + fDt, where
Dt = d t J (t) is the derivative of J with respect to t (Table 1,
1.2) and f is the learning rate. We approximated derivative
d t J (t) by the empirical average (Table 1, 1.3).

Phase 1: Interact with Robot and Collect Data

Policy
Generator

Action

1 Parameter

Policy

wn

State
Reward

Efficient Reuse of Previous Experiences

Importance Weight
The original PGPE can be considered an on-policy algorithm
[27], where the data collected from the current policy are used
to estimate the policy gradients. However, to reuse the previous experiences, we need to evaluate the current policy with
the data collected by the previous policies. To do this, we need
an off-policy algorithm through which the data-collecting policy and the policy to be updated are different. Therefore, we
use an off-policy version of the PGPE algorithm. In this offpolicy method, importance weighting [8] is used to evaluate
the previously collected data (experience)
from the current policy's point of view.
This method is called an IW-PGPE [16],
[24], [26], [34].
The basic idea of importance
weighting is to weight samples drawn
from a sampling distribution to match
Robot
the target distribution. For PGPE, importance weight v was defined for current hyperparameter t that was used
in previous experiences:

2 Repeat Until t = T

3 Repeat N Times

Phase 2: Add the Experienced Data to Database
Di
Add
Database

(b)
Phase 3: Update the Hyperparameter of a Policy

Policy
Generator

2 Dti

1t

3 vi

Di

4 Dt

Di-1

Di-L
Database

5 Repeat S Times
(c)
Figure 1. The flowchart of the proposed method: (a) Phase I, (b) Phase II, and
(c) Phase III.

98

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

march 2016

p (w l | t )
.
p (w l | t l )

(4)

This weight indicates how much the
previous experience contributed to the
current policy update. The approximated derivative of the expected return is
then weighted by this importance
weight for reusing the previous experiences. Table 1 shows the weighted derivatives using the previous experiences,
where w ln represents a policy parameter
generated from previous hyperparameter tl and hln represents the trajectory
of the previous experiences.

(a)

Hyperparameter
Generated Policy Parameter
Acquired Reward

v (w l ) =

Learning Procedure
The learning procedure of our proposed
method is shown in Figure 1, which repeats from Phases 1 to 3 until the learning performance is converged:
● Phase 1: Collect data in a real environment.
● Phase 2: Add the collected data to a
database.
● Phase 3: Update the hyperparameters
of the current policy using the stored
data in the database.
In Phase 1, 1 policy parameter w n is
sampled from prior distribution p (w | t).
2 Then, a trajectory is acquired from the
real environment using the policy with the
sampled policy parameter. In 3 , 1 and
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2016
