IEEE Systems, Man and Cybernetics Magazine - July 2019 - 26

Vstart

80
60
40
20

Vstop

0

0.8

Power

0.85
0.9
0.95
Voltage (p.u.)
(a)

1

Ppeak

tnow

tnow + thorizon
Time
(b)

Figure 5. Graphs showing the droop function of the

SoC (%)

local controller. (a) The function lowers the charging
power if there is an undervoltage issue. (b) The
droop function results in a power profile with a lower
peak (red); [7]. Vstart: the voltage level below which
the charging power will be linearly reduced; Vstop: the
voltage level below which the charging power will
be zero; p.u.: per unit, i.e., a reference to the nominal
voltage; Ppeak: power peak if droop function is not
applied; t now: the beginning of the control period;
t horizon: the duration of the control period.

100
80
60
40
20
0

10

20

30 40 50 60 70
Time (Quarter Hour)

80

90

behavior (e.g., opening doors or windows; Figure 6). Ideally,
the flexibility of all individual appliances should be modeled
before being aggregated to take full advantage of that flexibility. But that is impractical. A more practical method is to
identify "tracers" in the population of flexibility providers
and, in this way, obtain scalable approach [10]. A few tracers can track the population behavior of thousands of individual devices, at a much lower computational and
communicational cost. A cross-entropy method can be used
to identify such relevant tracers, and then apply the optimization step (step 2 of the three-step approach mentioned
previously) on the reduced order tracer models.
These tracers serve as examples of how machine learning can be added to the three-step approach to learn the
flexibility of a cluster of devices [10]. Vandael et al. have
applied machine learning to determine the EV cluster
behavior together with the energy market behavior. This
enabled them to optimally benefit from the flexibility on
the market [11].
Machine learning can also be applied to learn the flexibility of individual devices in a data-driven way. Ruelens
et al. applied this to an electric boiler, where boiler characteristics and market prices are considered together with user
behavior [12]. Based on fitted Q-iteration, the boiler directly
learns a control policy that determines when an electric boiler needs to charge, depending on its SoC, the time of day, and
the price of electricity (Figure 7). By exploring the design
space and exploiting earlier knowledge, it is able to determine a good control policy in fewer than 20 days.
Recently, Engels et al. [13] presented an example of using
flexibility for multiple purposes at the same time. They demonstrated a battery that, using stochastic optimization techniques, provides frequency support to the system operator
while also providing spare capacity for locally generated

SoC (%)

Power (%)

100

100
90
80
70
60
50
40
30

SoC (%)

Power (kW)

(a)
2.5
2
1.5
1
0.5
0

10

20

30 40 50 60 70
Time (Quarter Hour)

80

90

(b)
Figure 6. (a) A graph showing the SoC of 1,000

thermostatically controlled loads; physical differences
imply different dynamics. (b) A graph showing an
on-off signal as a straight black line and aggregated,
scaled power consumption as a dashed line [10].

26

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE Ju ly 2019

100
90
80
70
60
50
40
30

16

32
48
64
Time (Quarter Hour)
(a)

80

96

16

32
48
64
Time (Quarter Hour)
(b)

80

96

Figure 7. Graphs showing learned policy (a) and

smoothed policy (b) for boiler control, depending on
current SoC and price. Black: boiler on; white: boiler
off [12]. The red dotted line represents the price signal.
IEEE Systems, Man and Cybernetics Magazine - July 2019

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - July 2019
