IEEE Power & Energy Magazine - March/April 2018 - 78

example is for energy trading bidders to gain better positions or understanding for the next-day (or next-hour)
trading, based on the supervised learning and then the rl
in the policy network. the value network can be applied
to predict the benefit for candidate choices in bidding.
Further, the mctS or a similar algorithm can be used to
further evaluate the best potential actions for operators in
a control room or the optimal bidding for a trader in an
energy market.
in the next two sections, potential applications in power
systems are categorized into two groups based on the objectives of study, model interactivity, and complexity: game
based and search based.

figure 1. A sample Go game at the mid-game stage.

to gain further improvement of the policy network
through millions of self-played games. in actual games,
the policy network will select these expected moves based
on rl.
next, in the value network, evaluations of the winning
probability of candidate moves from the policy network
will be carried out. many convolutional layers will be employed to enhance the performance.
Finally, in the mctS, proper moves will be selected
using a look-ahead search. a combination of the mctS with
the policy and value networks is performed to eliminate
branches and leafs, that is, to reduce the breadth and depth in
the searching tree. Four steps (selection, expansion, evaluation, and backup) are performed in the mctS, as illustrated
in Figure 3.

Similarity Between Power
Systems AI and AlphaGo
While alphaGo is ai software designed
to mimic a human's thinking and logical reasoning, it presents some similarities with other power system problems.
For instance, alphaGo uses past human
expert games when training the neural network to form the policy network.
once the policy network is developed,
t he va lue net work ca n be a ch ieve d
with self-played games. this can be employed for power system problems in
identifying future weaknesses, for instance, with past data to form a policy
network. then, a large number of selfstudies based on simulations can be used
in combination with the previous policy
network to form a more accurate and enhanced value network. this can greatly
sharpen the knowledge base. another
78

ieee power & energy magazine

Potential Applications in
Game-Based Studies
this category includes many game-based problems in
power systems, such as operating economics, market
organization, and pricing, with more than one entity involved in those problems, such as generation companies
(Gencos), load-serving entities (lSe), distribution companies (diScos), and large consumers. these players act
as highly autonomous entities and interact with external
entities in ways that promotes their self-interests. the
framework of this type of problem is a noncooperative
game, which focuses on the competition among individual, rational players with a global equilibrium, usually
referred to as a nash equilibrium. as the whole system
gradually evolves to the final equilibrium state, each player has to strategically select his moves by considering the
possible moves of his opponents at multiple stages, which
shares much similarity with the game of Go and can be
potentially solved with an alphaGolike algorithm.

Strategic Wholesale
Market Bidding

figure 2. The cover page
of the Nature issue featuring
AlphaGo. (Reprinted with
permission from Macmillan
Publishers Ltd: Nature. Cover
page of vol. 529, copyright
2016.)

in a competitive wholesale electricity
market, the participants, including Gencos, diScos, and large consumers, submit their bids to the marker operator with
the aim of maximizing their profits. to
win the bidding, each participant has to
think of the strategies of its peer rivals.
Furthermore, in a multistage market that
includes day-ahead market, hour-ahead
market, and real-time market, the participants have to also think over all the possible future outcomes at each decisionmaking stage. these two features make
the bidding problem a natural fit for an
alphaGo-like algorithm.
the alphaGo algorithm can be applied
to strategic biddings in the following steps
(see Figure 4):
march/april 2018

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - March/April 2018