Instrumentation & Measurement Magazine 24-2 - 121

function is selected to be positive definite [18]. The following
development explains how the Value Iteration process approximates the solving value function Vℓ and the optimal strategy uℓ
using means of the adaptive actor-critics.

pursuer only depend on its own location measurements and
those of the evader ship.

Adaptive Actor-Critic Approximation

Total::Perception Engine and Coordinate
Transformations

In this work, a critic neural network is applied to approximate the solving value function, whereas an actor neural
network is designed to estimate the optimal strategy [19],
 to denote an
[20]. In the following, we will use the notation (·)
estimate of (·).
The approximation of the solving value function for each
pursuing vessel is motivated by the following quadratic strucwhere   { x , y}:
ture in the  -direction,
 {x , y}
	

T
1
Vˆ  ( Ek , uˆ k )   E k
2

E 
T
uˆ k   W [ c ]  k  ,	

 uˆ k 

with W [ c ]   44 being the critic approximation weights matrix
of the following block structure
	

[ c ]
W [ c ] WEu

W [ c ]   EE
.	
[ c ]
[ c ] 
W
W

uu 
 uE

In a similar fashion, the optimized strategy is estimated by
a form similar to that of (2) and (6),
uˆ k  W [ a ] Ek ,	

	

where W [ a ]   13 holds the actor approximation weights.
Let  k[ c ] and  k[ a ] be critic and actor weight approximation errors, respectively. Then,

	

2
1 ˆ  
V (E k , uˆ k )  V k 
	
2
2
1 
[ a ]

uˆ  u k 

k

2 k


 k[ c ]

Simulation Scenarios

While the decision-making mechanism is realized through
the RL technique listed in Algorithm 1, the vessels' dynamics and the evader ship behavior are simulated using
Total::Perception™ (T::P), which is a state-of-the-art proprietary Intelligence, Surveillance and Reconnaissance (ISR)
Systems Simulation Engine developed by Larus Technologies.
It is capable of fusing large amounts of data from a multitude of sources, such as, active and passive sensing packages
from GPS, Radar, and other sources. It is also able to learn
complex relations among the tracked vessels to recognize
any anomalous patterns in order to provide accurate information-processing strategies and continuously optimize the
situational awareness mechanisms for decision support makers. Sea vessel objects in T::P are controlled through a variety
of commands, such as the vessel's reference linear speed and
orientation angle. The engine also enables chosen vessels to
perform different missions like patrolling and interception. To
assess the performance of the proposed solution, the heading
angles and the linear speeds of the pursuer vessels are controlled by the RL approach described in Algorithm 1, while the
evader's intelligence is provided by T::P.
Let RF0 be the (XYZ) Earth Centered Earth Fixed (ECEF) coordinates, as shown in Figure 1. The T::P engine operates with
spherical coordinates in RF0. If the spherical coordinates of a
vector u in RF0 are ( u , ,  ), where u stands for the Euclidean
norm, and θ and γ are the longitude and latitude angles, respectively, then the Cartesian coordinates of u in RF0 are given by

where
	





The update of the critic and actor weights is accomplished
in real-time through a gradient descent approach, such that
	

	

E 
T
[ c ] ( r 1)
W
W [ c ]( r )   c  k[ c ]  k   E k

ˆ
 uk 

T
uˆ k 	(8)


[ a ] ( r 1)
W
W [ a ] ( r )   a k[ a ] E k	(9)

where 0 < αc, αa < 1 are learning rate parameters and r is the index corresponding to the strategy update loop. The adaptive
actor-critic implementation of the online Value Iteration process is detailed in Algorithm 1. The matrix magnitude used in
the algorithm is defined as the maximum absolute value of the
matrix elements. It is important to notice that the designed RL
algorithm does not use any information about the dynamics of
the pursuer or evader vessels. The strategies followed by each
April 2021	

0

	


V k U  E k , uˆ k  Vˆ  ( E k1 , uˆ k 1 )
	
u k   (Wuu[ c ] )1WuE[ c ]  E k

cos( )cos( ) 


u  u  cos( )sin( ) 	
 sin( ) 



Converting this to its equivalent Cartesian coordinates ju
in the (ENU) frame RFj corresponding to vessel j is performed
j
j
j
0
through the rotation matrix Ri, where u  R0 u and

	

j

  sin( )
cos( )
0 






 ) 	
R0 

cos(
)
sin
(
)

sin(
)sin(
)
cos(

 cos( ) cos( )
sin( ) cos( ) sin( ) 


To go back to RF0, it is sufficient to multiply by the transpose of
T
R0 (a property of rotation matrices). In other words, 0 u  j R0 j u.
For each pursuer j, the RL process generates the heading angle and surge speed variations or control decisions for that
pursuer expressed in RFj. They are transformed into their
equivalent Cartesian coordinates in RF0 before passing them
on to the T::P engine. The updated vessel poses generated by
T::P are fed back to the RL algorithm after they are transformed
back to their respective vessel frame RFj coordinates. The T::P
j

 

IEEE Instrumentation & Measurement Magazine	121



Instrumentation & Measurement Magazine 24-2

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-2

No label
Instrumentation & Measurement Magazine 24-2 - No label
Instrumentation & Measurement Magazine 24-2 - Cover2
Instrumentation & Measurement Magazine 24-2 - 1
Instrumentation & Measurement Magazine 24-2 - 2
Instrumentation & Measurement Magazine 24-2 - 3
Instrumentation & Measurement Magazine 24-2 - 4
Instrumentation & Measurement Magazine 24-2 - 5
Instrumentation & Measurement Magazine 24-2 - 6
Instrumentation & Measurement Magazine 24-2 - 7
Instrumentation & Measurement Magazine 24-2 - 8
Instrumentation & Measurement Magazine 24-2 - 9
Instrumentation & Measurement Magazine 24-2 - 10
Instrumentation & Measurement Magazine 24-2 - 11
Instrumentation & Measurement Magazine 24-2 - 12
Instrumentation & Measurement Magazine 24-2 - 13
Instrumentation & Measurement Magazine 24-2 - 14
Instrumentation & Measurement Magazine 24-2 - 15
Instrumentation & Measurement Magazine 24-2 - 16
Instrumentation & Measurement Magazine 24-2 - 17
Instrumentation & Measurement Magazine 24-2 - 18
Instrumentation & Measurement Magazine 24-2 - 19
Instrumentation & Measurement Magazine 24-2 - 20
Instrumentation & Measurement Magazine 24-2 - 21
Instrumentation & Measurement Magazine 24-2 - 22
Instrumentation & Measurement Magazine 24-2 - 23
Instrumentation & Measurement Magazine 24-2 - 24
Instrumentation & Measurement Magazine 24-2 - 25
Instrumentation & Measurement Magazine 24-2 - 26
Instrumentation & Measurement Magazine 24-2 - 27
Instrumentation & Measurement Magazine 24-2 - 28
Instrumentation & Measurement Magazine 24-2 - 29
Instrumentation & Measurement Magazine 24-2 - 30
Instrumentation & Measurement Magazine 24-2 - 31
Instrumentation & Measurement Magazine 24-2 - 32
Instrumentation & Measurement Magazine 24-2 - 33
Instrumentation & Measurement Magazine 24-2 - 34
Instrumentation & Measurement Magazine 24-2 - 35
Instrumentation & Measurement Magazine 24-2 - 36
Instrumentation & Measurement Magazine 24-2 - 37
Instrumentation & Measurement Magazine 24-2 - 38
Instrumentation & Measurement Magazine 24-2 - 39
Instrumentation & Measurement Magazine 24-2 - 40
Instrumentation & Measurement Magazine 24-2 - 41
Instrumentation & Measurement Magazine 24-2 - 42
Instrumentation & Measurement Magazine 24-2 - 43
Instrumentation & Measurement Magazine 24-2 - 44
Instrumentation & Measurement Magazine 24-2 - 45
Instrumentation & Measurement Magazine 24-2 - 46
Instrumentation & Measurement Magazine 24-2 - 47
Instrumentation & Measurement Magazine 24-2 - 48
Instrumentation & Measurement Magazine 24-2 - 49
Instrumentation & Measurement Magazine 24-2 - 50
Instrumentation & Measurement Magazine 24-2 - 51
Instrumentation & Measurement Magazine 24-2 - 52
Instrumentation & Measurement Magazine 24-2 - 53
Instrumentation & Measurement Magazine 24-2 - 54
Instrumentation & Measurement Magazine 24-2 - 55
Instrumentation & Measurement Magazine 24-2 - 56
Instrumentation & Measurement Magazine 24-2 - 57
Instrumentation & Measurement Magazine 24-2 - 58
Instrumentation & Measurement Magazine 24-2 - 59
Instrumentation & Measurement Magazine 24-2 - 60
Instrumentation & Measurement Magazine 24-2 - 61
Instrumentation & Measurement Magazine 24-2 - 62
Instrumentation & Measurement Magazine 24-2 - 63
Instrumentation & Measurement Magazine 24-2 - 64
Instrumentation & Measurement Magazine 24-2 - 65
Instrumentation & Measurement Magazine 24-2 - 66
Instrumentation & Measurement Magazine 24-2 - 67
Instrumentation & Measurement Magazine 24-2 - 68
Instrumentation & Measurement Magazine 24-2 - 69
Instrumentation & Measurement Magazine 24-2 - 70
Instrumentation & Measurement Magazine 24-2 - 71
Instrumentation & Measurement Magazine 24-2 - 72
Instrumentation & Measurement Magazine 24-2 - 73
Instrumentation & Measurement Magazine 24-2 - 74
Instrumentation & Measurement Magazine 24-2 - 75
Instrumentation & Measurement Magazine 24-2 - 76
Instrumentation & Measurement Magazine 24-2 - 77
Instrumentation & Measurement Magazine 24-2 - 78
Instrumentation & Measurement Magazine 24-2 - 79
Instrumentation & Measurement Magazine 24-2 - 80
Instrumentation & Measurement Magazine 24-2 - 81
Instrumentation & Measurement Magazine 24-2 - 82
Instrumentation & Measurement Magazine 24-2 - 83
Instrumentation & Measurement Magazine 24-2 - 84
Instrumentation & Measurement Magazine 24-2 - 85
Instrumentation & Measurement Magazine 24-2 - 86
Instrumentation & Measurement Magazine 24-2 - 87
Instrumentation & Measurement Magazine 24-2 - 88
Instrumentation & Measurement Magazine 24-2 - 89
Instrumentation & Measurement Magazine 24-2 - 90
Instrumentation & Measurement Magazine 24-2 - 91
Instrumentation & Measurement Magazine 24-2 - 92
Instrumentation & Measurement Magazine 24-2 - 93
Instrumentation & Measurement Magazine 24-2 - 94
Instrumentation & Measurement Magazine 24-2 - 95
Instrumentation & Measurement Magazine 24-2 - 96
Instrumentation & Measurement Magazine 24-2 - 97
Instrumentation & Measurement Magazine 24-2 - 98
Instrumentation & Measurement Magazine 24-2 - 99
Instrumentation & Measurement Magazine 24-2 - 100
Instrumentation & Measurement Magazine 24-2 - 101
Instrumentation & Measurement Magazine 24-2 - 102
Instrumentation & Measurement Magazine 24-2 - 103
Instrumentation & Measurement Magazine 24-2 - 104
Instrumentation & Measurement Magazine 24-2 - 105
Instrumentation & Measurement Magazine 24-2 - 106
Instrumentation & Measurement Magazine 24-2 - 107
Instrumentation & Measurement Magazine 24-2 - 108
Instrumentation & Measurement Magazine 24-2 - 109
Instrumentation & Measurement Magazine 24-2 - 110
Instrumentation & Measurement Magazine 24-2 - 111
Instrumentation & Measurement Magazine 24-2 - 112
Instrumentation & Measurement Magazine 24-2 - 113
Instrumentation & Measurement Magazine 24-2 - 114
Instrumentation & Measurement Magazine 24-2 - 115
Instrumentation & Measurement Magazine 24-2 - 116
Instrumentation & Measurement Magazine 24-2 - 117
Instrumentation & Measurement Magazine 24-2 - 118
Instrumentation & Measurement Magazine 24-2 - 119
Instrumentation & Measurement Magazine 24-2 - 120
Instrumentation & Measurement Magazine 24-2 - 121
Instrumentation & Measurement Magazine 24-2 - 122
Instrumentation & Measurement Magazine 24-2 - 123
Instrumentation & Measurement Magazine 24-2 - 124
Instrumentation & Measurement Magazine 24-2 - 125
Instrumentation & Measurement Magazine 24-2 - 126
Instrumentation & Measurement Magazine 24-2 - 127
Instrumentation & Measurement Magazine 24-2 - 128
Instrumentation & Measurement Magazine 24-2 - 129
Instrumentation & Measurement Magazine 24-2 - 130
Instrumentation & Measurement Magazine 24-2 - 131
Instrumentation & Measurement Magazine 24-2 - 132
Instrumentation & Measurement Magazine 24-2 - Cover3
Instrumentation & Measurement Magazine 24-2 - Cover4
https://www.nxtbook.com/allen/iamm/25-9
https://www.nxtbook.com/allen/iamm/25-8
https://www.nxtbook.com/allen/iamm/25-7
https://www.nxtbook.com/allen/iamm/25-6
https://www.nxtbook.com/allen/iamm/25-5
https://www.nxtbook.com/allen/iamm/25-4
https://www.nxtbook.com/allen/iamm/25-3
https://www.nxtbook.com/allen/iamm/instrumentation-measurement-magazine-25-2
https://www.nxtbook.com/allen/iamm/25-1
https://www.nxtbook.com/allen/iamm/24-9
https://www.nxtbook.com/allen/iamm/24-7
https://www.nxtbook.com/allen/iamm/24-8
https://www.nxtbook.com/allen/iamm/24-6
https://www.nxtbook.com/allen/iamm/24-5
https://www.nxtbook.com/allen/iamm/24-4
https://www.nxtbook.com/allen/iamm/24-3
https://www.nxtbook.com/allen/iamm/24-2
https://www.nxtbook.com/allen/iamm/24-1
https://www.nxtbook.com/allen/iamm/23-9
https://www.nxtbook.com/allen/iamm/23-8
https://www.nxtbook.com/allen/iamm/23-6
https://www.nxtbook.com/allen/iamm/23-5
https://www.nxtbook.com/allen/iamm/23-2
https://www.nxtbook.com/allen/iamm/23-3
https://www.nxtbook.com/allen/iamm/23-4
https://www.nxtbookmedia.com