IEEE Power & Energy Magazine - January/February 2020 - 74

regulators, and smart inverters as RL agents that learn to make
better decisions using their local measurements and the his-
tory of past decisions. These can be online learning agents that
start with a base control policy and improve based on con-
tinuous interaction with their environment, i.e., the distribu-
tion system. The agent can learn a deterministic or stochastic
policy, where policy is a mapping for the agent's action to each
system state it encounters. An RL agent essentially maximizes
discounted rewards that depend upon its current and future
rewards when following the current control policy. Theoreti-
cally, an optimal control policy can be learned by solving Bell-
man optimality equations defined for a state- or action-value
function. This problem can be overcome using offline plan-
ning methods such as value iteration and policy iteration algo-
rithms or online learning methods, e.g., Q-learning. Another
class of algorithm more suited for problems with large state
space (such as power systems) is policy gradients, where the
agent optimizes the policy directly. The policy is modeled as
a parameterized function often approximated using universal
function approximators, such as neural networks.
In such settings, ADMSs can play a supervisory role and/
or act as a coordinator for the numerous local autonomous
decision-making agents. For example, ADMSs can drive
the learning of the local RL agents by providing them with
appropriate reward functions that help meet some system-
wide objectives. Centralized training with decentralized exe-
cution, a popular concept in multiagent RL, can be applied
to motivate cooperative behavior from the local autonomous
agents. An example of RL-based machine learning used to
induce cooperative behavior from multiple voltage control
devices is displayed in Figure 7. For this setting, even if the
centralized ADMS fails or is rendered unresponsive, the
local RL agents continue to make near-optimal decisions
using their training based on the prior data.
For load-response applications, an ADMS can act as a coor-
dinator by sending a global objective signal to encourage
cooperative decision making for dispersed controllable loads,
such as electric vehicles (EVs). As an example, an ADMS

Substation

can send a signal indicating the required decrease in load
demand during peak load hours to all EVs. Multiple EVs may
coordinate the decrease in their power consumption to col-
laboratively meet the peak reduction requirement. These are
just a few examples of using intelligence-based concepts in
decision making for power distribution systems. The poten-
tial applications of the new and emerging machine-learning
algorithms are endless and will continue to evolve with the
complexity of power distribution systems.

Data Quality and Consistency
The massive installation of information and communications
technology for remote control and the monitoring of distribu-
tion feeders may lead to increased vulnerabilities with respect
to cyberintrusions and false-data injections. Often, simple
resource-constrained sensors such as smart meters or field
devices are the first line of attack, rendering the data unreli-
able. Automatic data calibration modules, which can detect
such data anomalies and mitigate them by learning the most
probable measurement values using historical data analysis,
are called for. There is also a critical need to analyze the data
flow in, out, and within the ADMS and detect possible cases
of anomalies such as corrupted data, unnecessary data flows,
and any other malicious cyberactivities. Future ADMS appli-
cations will benefit from purely data-centric components that
specifically aim to quantify and improve the quality and con-
sistency of distribution grids' operational data.
The requirement for local sensor data calibration modules
is being recognized to process local data, identify any anomaly,
and apply corrections to improve data quality. The key concept
is to leverage data science and machine-learning algorithms to
develop models for time-series measurements and use those to
predict correct measurement values. Independent data quality
modules can also be developed for each measurement sensor
such as smart meters, micro-phasor measurement unit, and
field devices. Improving data quality at the local sensor level
does not require network-wide information and can be easily
integrated by collocating these modules at the respective sensor

Voltage Regulator

Load
Capacitor
Bank
ADMS

Communicate Reference Trajectory or a Global
Reward Signal (Substation Power Flow)

Smart Inverter

Autonomous Decision-Making Agents
Using the RL Method

figure 7. Autonomous decision making coordinated using an ADMS.
74

ieee power & energy magazine

january/february 2020
IEEE Power & Energy Magazine - January/February 2020

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - January/February 2020
