Instrumentation & Measurement Magazine 24-2 - 119

artificial intelligence algorithms on integrated microproces- tangent to the earth surface, which is modeled as a sphere
sors [6]. Such a technological breakthrough opened a new of radius R (see Figure 1). Let i(·) denote the quantity (·) exwindow of engineering application opportunities that were pressed in RFi; and ( i xkj , i y kj )   2 , i vkj  , and ikj  SO(2), be
not possible a few years ago. This includes, for example, the the x-y coordinates, linear surge speed (along Ej), and the
visual detection and autonomous refueling of UAVs [7], ro- orientation angle, respectively, of vessel j expressed in RFi,
bot motion planing [8], brain MRI estimation methods [9], where j = o, 1, 2, ..., N. Vessel o denotes the evader, whereas
and latency measurement of distributed networks [10], to vessels 1, 2, ..., N, denote the pursuers. The subscript k and
name a few. In a particular context, Approximate Dynamic SO(2) refer to the discrete-time index and the Special OrProgramming (ADP) is a machine learning tool that has been thogonal group in 2D, respectively. The decision-making
employed to tackle the computational challenges associated mechanism is executed in a decentralized fashion where the
with the action and state spaces of the dynamic programming RL algorithm is run locally by each pursuer j = 1, 2, ..., N,
solutions. ADP can be classified into four classes, namely, with the goal of minimizing the magnitudes of the tracking
,j
,j
Heuristic Dynamic Programming, Dual Heuristic Dynamic errors e k
 kj   ko,
 jk   ok, for   { x , y}, and consequently ek 
Programming, Action Dependent Heuristic Dynamic Pro- as k  . This is done by computing two adjustment scagramming, and Action Dependent Dual Heuristic Dynamic lars ukx , j and uky , j which are used to tune the surge velocity and
Programming. These classes are used to solve the Hamil- heading of vessel j, such that
ton-Jacobi (HJ) or Hamilton-Jacobi-Bellman (HJB) equations
j j
j j
vk 1 
vk   (ukv , j  ukv, j1 )	
for different optimization problems for single and multi- 	
agent systems [11]-[13]. This is done in order to optimize a
j j
j j
 k
 k  uk , j	
utility function associated to the dynamic system under con- 	
1
sideration. Devising solutions for multi-agent optimization
problems can be computationally challenging due to the cou- where 	
ukv , j
(ukx , j )2 
 (uky , j )2 , uk , j tan 1 (ukx , j / uky , j )  [ / 2,  / 2] rad,
pling effects in the optimality equations [14]. Furthermore, the
complexity increases even further when trying to solve that and δ is a discount factor. Since the same RL algorithm is run
problem in a distributed fashion using only local information by each vessel j and the output signals are computed with respect to RFj, the reference frame and the vessel identifiers will
available to each individual agent.
Reinforcement Learning (RL) is a ML mechanism that is no longer be explicitly indicated in the rest of the manuscript.
The corrective adjustments uk ,   { x , y}, are set up as
adaptive to the different ADP solutions [15]. It is concerned
with selecting the best strategy-to-apply in a dynamic learning
process that penalizes or rewards such strategy using a given 	
uk  K k E k	(2)
cost function. This enables the system to transit from one state
T




3
to another which represents a better choice in the path of find- where Ek   e k e k1 e k2    is a window of the last
ing an optimal solution to the problem in hand [16]. The RL three recorded tracking errors between the pursuer and the
process mimics the analytical solution environment of the optimization problems where the optimal strategy and its value
are interrelated [11]. The RL platform employs temporal difference solutions arising from the various ADP classes. However,
these solutions may get overwhelmingly complex especially in
the case of multi-agent systems. Hence, solvers that are based
on combined ideas of game theory, cooperative optimization,
and RL are used to provide distributed and real-time solutions
for such problems. The RL solutions are implemented using
one of two two-step techniques known as value iteration and
policy iteration methods. They differ at how the strategy is
evaluated and then updated accordingly [15]. Within the same
context, the adaptive actor-critic structures are approximation
tools used by the RL algorithms. The actor is a neural network
that approximates the adopted policy, while the critic is a neural network that estimates the cost-to-go from each state to a
final following a certain policy [17].

Pursuer-Evader Problem Setup
Consider a system of one evader and N pursuers. Each
vessel j is associated with an (ENU) Cartesian coordinate
system RFj pinned at the vessel's position, such that vector
Ej points to the vessel's heading and the 2D plane (Ej, Nj) is
April 2021	

Fig. 1. Earth and vessel coordinate systems.

IEEE Instrumentation & Measurement Magazine	119



Instrumentation & Measurement Magazine 24-2

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-2

No label
Instrumentation & Measurement Magazine 24-2 - No label
Instrumentation & Measurement Magazine 24-2 - Cover2
Instrumentation & Measurement Magazine 24-2 - 1
Instrumentation & Measurement Magazine 24-2 - 2
Instrumentation & Measurement Magazine 24-2 - 3
Instrumentation & Measurement Magazine 24-2 - 4
Instrumentation & Measurement Magazine 24-2 - 5
Instrumentation & Measurement Magazine 24-2 - 6
Instrumentation & Measurement Magazine 24-2 - 7
Instrumentation & Measurement Magazine 24-2 - 8
Instrumentation & Measurement Magazine 24-2 - 9
Instrumentation & Measurement Magazine 24-2 - 10
Instrumentation & Measurement Magazine 24-2 - 11
Instrumentation & Measurement Magazine 24-2 - 12
Instrumentation & Measurement Magazine 24-2 - 13
Instrumentation & Measurement Magazine 24-2 - 14
Instrumentation & Measurement Magazine 24-2 - 15
Instrumentation & Measurement Magazine 24-2 - 16
Instrumentation & Measurement Magazine 24-2 - 17
Instrumentation & Measurement Magazine 24-2 - 18
Instrumentation & Measurement Magazine 24-2 - 19
Instrumentation & Measurement Magazine 24-2 - 20
Instrumentation & Measurement Magazine 24-2 - 21
Instrumentation & Measurement Magazine 24-2 - 22
Instrumentation & Measurement Magazine 24-2 - 23
Instrumentation & Measurement Magazine 24-2 - 24
Instrumentation & Measurement Magazine 24-2 - 25
Instrumentation & Measurement Magazine 24-2 - 26
Instrumentation & Measurement Magazine 24-2 - 27
Instrumentation & Measurement Magazine 24-2 - 28
Instrumentation & Measurement Magazine 24-2 - 29
Instrumentation & Measurement Magazine 24-2 - 30
Instrumentation & Measurement Magazine 24-2 - 31
Instrumentation & Measurement Magazine 24-2 - 32
Instrumentation & Measurement Magazine 24-2 - 33
Instrumentation & Measurement Magazine 24-2 - 34
Instrumentation & Measurement Magazine 24-2 - 35
Instrumentation & Measurement Magazine 24-2 - 36
Instrumentation & Measurement Magazine 24-2 - 37
Instrumentation & Measurement Magazine 24-2 - 38
Instrumentation & Measurement Magazine 24-2 - 39
Instrumentation & Measurement Magazine 24-2 - 40
Instrumentation & Measurement Magazine 24-2 - 41
Instrumentation & Measurement Magazine 24-2 - 42
Instrumentation & Measurement Magazine 24-2 - 43
Instrumentation & Measurement Magazine 24-2 - 44
Instrumentation & Measurement Magazine 24-2 - 45
Instrumentation & Measurement Magazine 24-2 - 46
Instrumentation & Measurement Magazine 24-2 - 47
Instrumentation & Measurement Magazine 24-2 - 48
Instrumentation & Measurement Magazine 24-2 - 49
Instrumentation & Measurement Magazine 24-2 - 50
Instrumentation & Measurement Magazine 24-2 - 51
Instrumentation & Measurement Magazine 24-2 - 52
Instrumentation & Measurement Magazine 24-2 - 53
Instrumentation & Measurement Magazine 24-2 - 54
Instrumentation & Measurement Magazine 24-2 - 55
Instrumentation & Measurement Magazine 24-2 - 56
Instrumentation & Measurement Magazine 24-2 - 57
Instrumentation & Measurement Magazine 24-2 - 58
Instrumentation & Measurement Magazine 24-2 - 59
Instrumentation & Measurement Magazine 24-2 - 60
Instrumentation & Measurement Magazine 24-2 - 61
Instrumentation & Measurement Magazine 24-2 - 62
Instrumentation & Measurement Magazine 24-2 - 63
Instrumentation & Measurement Magazine 24-2 - 64
Instrumentation & Measurement Magazine 24-2 - 65
Instrumentation & Measurement Magazine 24-2 - 66
Instrumentation & Measurement Magazine 24-2 - 67
Instrumentation & Measurement Magazine 24-2 - 68
Instrumentation & Measurement Magazine 24-2 - 69
Instrumentation & Measurement Magazine 24-2 - 70
Instrumentation & Measurement Magazine 24-2 - 71
Instrumentation & Measurement Magazine 24-2 - 72
Instrumentation & Measurement Magazine 24-2 - 73
Instrumentation & Measurement Magazine 24-2 - 74
Instrumentation & Measurement Magazine 24-2 - 75
Instrumentation & Measurement Magazine 24-2 - 76
Instrumentation & Measurement Magazine 24-2 - 77
Instrumentation & Measurement Magazine 24-2 - 78
Instrumentation & Measurement Magazine 24-2 - 79
Instrumentation & Measurement Magazine 24-2 - 80
Instrumentation & Measurement Magazine 24-2 - 81
Instrumentation & Measurement Magazine 24-2 - 82
Instrumentation & Measurement Magazine 24-2 - 83
Instrumentation & Measurement Magazine 24-2 - 84
Instrumentation & Measurement Magazine 24-2 - 85
Instrumentation & Measurement Magazine 24-2 - 86
Instrumentation & Measurement Magazine 24-2 - 87
Instrumentation & Measurement Magazine 24-2 - 88
Instrumentation & Measurement Magazine 24-2 - 89
Instrumentation & Measurement Magazine 24-2 - 90
Instrumentation & Measurement Magazine 24-2 - 91
Instrumentation & Measurement Magazine 24-2 - 92
Instrumentation & Measurement Magazine 24-2 - 93
Instrumentation & Measurement Magazine 24-2 - 94
Instrumentation & Measurement Magazine 24-2 - 95
Instrumentation & Measurement Magazine 24-2 - 96
Instrumentation & Measurement Magazine 24-2 - 97
Instrumentation & Measurement Magazine 24-2 - 98
Instrumentation & Measurement Magazine 24-2 - 99
Instrumentation & Measurement Magazine 24-2 - 100
Instrumentation & Measurement Magazine 24-2 - 101
Instrumentation & Measurement Magazine 24-2 - 102
Instrumentation & Measurement Magazine 24-2 - 103
Instrumentation & Measurement Magazine 24-2 - 104
Instrumentation & Measurement Magazine 24-2 - 105
Instrumentation & Measurement Magazine 24-2 - 106
Instrumentation & Measurement Magazine 24-2 - 107
Instrumentation & Measurement Magazine 24-2 - 108
Instrumentation & Measurement Magazine 24-2 - 109
Instrumentation & Measurement Magazine 24-2 - 110
Instrumentation & Measurement Magazine 24-2 - 111
Instrumentation & Measurement Magazine 24-2 - 112
Instrumentation & Measurement Magazine 24-2 - 113
Instrumentation & Measurement Magazine 24-2 - 114
Instrumentation & Measurement Magazine 24-2 - 115
Instrumentation & Measurement Magazine 24-2 - 116
Instrumentation & Measurement Magazine 24-2 - 117
Instrumentation & Measurement Magazine 24-2 - 118
Instrumentation & Measurement Magazine 24-2 - 119
Instrumentation & Measurement Magazine 24-2 - 120
Instrumentation & Measurement Magazine 24-2 - 121
Instrumentation & Measurement Magazine 24-2 - 122
Instrumentation & Measurement Magazine 24-2 - 123
Instrumentation & Measurement Magazine 24-2 - 124
Instrumentation & Measurement Magazine 24-2 - 125
Instrumentation & Measurement Magazine 24-2 - 126
Instrumentation & Measurement Magazine 24-2 - 127
Instrumentation & Measurement Magazine 24-2 - 128
Instrumentation & Measurement Magazine 24-2 - 129
Instrumentation & Measurement Magazine 24-2 - 130
Instrumentation & Measurement Magazine 24-2 - 131
Instrumentation & Measurement Magazine 24-2 - 132
Instrumentation & Measurement Magazine 24-2 - Cover3
Instrumentation & Measurement Magazine 24-2 - Cover4
https://www.nxtbook.com/allen/iamm/25-9
https://www.nxtbook.com/allen/iamm/25-8
https://www.nxtbook.com/allen/iamm/25-7
https://www.nxtbook.com/allen/iamm/25-6
https://www.nxtbook.com/allen/iamm/25-5
https://www.nxtbook.com/allen/iamm/25-4
https://www.nxtbook.com/allen/iamm/25-3
https://www.nxtbook.com/allen/iamm/instrumentation-measurement-magazine-25-2
https://www.nxtbook.com/allen/iamm/25-1
https://www.nxtbook.com/allen/iamm/24-9
https://www.nxtbook.com/allen/iamm/24-7
https://www.nxtbook.com/allen/iamm/24-8
https://www.nxtbook.com/allen/iamm/24-6
https://www.nxtbook.com/allen/iamm/24-5
https://www.nxtbook.com/allen/iamm/24-4
https://www.nxtbook.com/allen/iamm/24-3
https://www.nxtbook.com/allen/iamm/24-2
https://www.nxtbook.com/allen/iamm/24-1
https://www.nxtbook.com/allen/iamm/23-9
https://www.nxtbook.com/allen/iamm/23-8
https://www.nxtbook.com/allen/iamm/23-6
https://www.nxtbook.com/allen/iamm/23-5
https://www.nxtbook.com/allen/iamm/23-2
https://www.nxtbook.com/allen/iamm/23-3
https://www.nxtbook.com/allen/iamm/23-4
https://www.nxtbookmedia.com