Instrumentation & Measurement Magazine 25-6 - 26

for the great majority of network applications. For more details
about the lab results, please see [4].
In the Real World: Deploying Deme
As mentioned at the beginning, one of the challenges with ML
systems is that the distribution of the data in the real world
might be different from the lab data with which the ML system
was trained [6]. More specifically, although the distribution of
the input to the system might not change (Deme takes source
and destination nodes' IP addresses as input, and the distribution
of those have not really changed since 2002 when KING
was collected), the conditional distribution of the output (in
Deme's case the RTT values) might change. This phenomenon
is well known in ML since the 1980s [7] and is referred to
as concept drift.
Our industry partner Swarmio Inc., who eventually liFig.
2. The design of the AI model, which is a Multimodal CNN. ©IEEE,
reused with permission from [4].
feature. Finally, all features are normalized and sent to the
AI model, which is shown in Fig. 2. The AI model combines
two CNN models, one for nodes within the same continent
(local model) and another for nodes from different continents
(continental model), followed by fully connected
layers to get the best possible performance. The CNN itself
is shown in Fig. 3, and the AI model's design parameters are
listed in Table 1.
For training, the KING [5] dataset was used, because KING
is the largest publicly available network delay measurement
dataset, consisting of about 100 million pairwise Round-Trip
Time (RTT) measurements from 1740 DNS servers. In lab tests,
Deme was able to predict RTT values between any two IP addresses
with an average accuracy of 96.1%, which is sufficient
censed and patented Deme [8], needed to be sure that in the
real world, Deme will work with acceptable performance,
which they defined as 90% or more accuracy. To demonstrate
that, we used MLOps [9] to deploy Deme in a real setting. As
illustrated by Fig. 4, MLOps integrates ML with software engineering's
well-known concept of Development Operations
(DevOps), which in turn is a methodology for developing,
deploying, and maintaining large-scale software systems.
Similar to DevOps, MLOps ensures Continuous Integration
(CI) and Continuous Delivery (CD). In addition, MLOps also
adds the concept of Continuous Training (CT). At the beginning
and in the ML stage (Fig. 4, top), a dataset is prepared and
used to train, validate and test the ML model in a lab. If the
model successfully passes the test, it moves to the Ops stage
(Fig. 4, bottom) where it undergoes the usual software engineering's
unit and integration testing. If it also passes those
tests, it is packaged with the rest of the system and deployed.
While deployed, the model is continuously monitored and
compared to a baseline reference. Should performance become
unsatisfactory, for example the accuracy goes below a predefined
acceptable threshold, CT is triggered and the model
is sent back to the ML stage with new data gathered from the
field. At the CT stage, the model must be refined to give a better
performance with the new data.
In our case, we deployed Deme in an actual Swarmiohosted
gaming tournament consisting of 189 players connected
to nine servers. We measured the actual RTT values as follows.
Fig. 3. Each CNN model in the Multimodal CNN. ©IEEE, reused with permission from [4].
26
IEEE Instrumentation & Measurement Magazine
September 2022

Instrumentation & Measurement Magazine 25-6

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 25-6

Instrumentation & Measurement Magazine 25-6 - Cover1
Instrumentation & Measurement Magazine 25-6 - Cover2
Instrumentation & Measurement Magazine 25-6 - 1
Instrumentation & Measurement Magazine 25-6 - 2
Instrumentation & Measurement Magazine 25-6 - 3
Instrumentation & Measurement Magazine 25-6 - 4
Instrumentation & Measurement Magazine 25-6 - 5
Instrumentation & Measurement Magazine 25-6 - 6
Instrumentation & Measurement Magazine 25-6 - 7
Instrumentation & Measurement Magazine 25-6 - 8
Instrumentation & Measurement Magazine 25-6 - 9
Instrumentation & Measurement Magazine 25-6 - 10
Instrumentation & Measurement Magazine 25-6 - 11
Instrumentation & Measurement Magazine 25-6 - 12
Instrumentation & Measurement Magazine 25-6 - 13
Instrumentation & Measurement Magazine 25-6 - 14
Instrumentation & Measurement Magazine 25-6 - 15
Instrumentation & Measurement Magazine 25-6 - 16
Instrumentation & Measurement Magazine 25-6 - 17
Instrumentation & Measurement Magazine 25-6 - 18
Instrumentation & Measurement Magazine 25-6 - 19
Instrumentation & Measurement Magazine 25-6 - 20
Instrumentation & Measurement Magazine 25-6 - 21
Instrumentation & Measurement Magazine 25-6 - 22
Instrumentation & Measurement Magazine 25-6 - 23
Instrumentation & Measurement Magazine 25-6 - 24
Instrumentation & Measurement Magazine 25-6 - 25
Instrumentation & Measurement Magazine 25-6 - 26
Instrumentation & Measurement Magazine 25-6 - 27
Instrumentation & Measurement Magazine 25-6 - 28
Instrumentation & Measurement Magazine 25-6 - 29
Instrumentation & Measurement Magazine 25-6 - 30
Instrumentation & Measurement Magazine 25-6 - 31
Instrumentation & Measurement Magazine 25-6 - 32
Instrumentation & Measurement Magazine 25-6 - 33
Instrumentation & Measurement Magazine 25-6 - 34
Instrumentation & Measurement Magazine 25-6 - 35
Instrumentation & Measurement Magazine 25-6 - 36
Instrumentation & Measurement Magazine 25-6 - 37
Instrumentation & Measurement Magazine 25-6 - 38
Instrumentation & Measurement Magazine 25-6 - 39
Instrumentation & Measurement Magazine 25-6 - 40
Instrumentation & Measurement Magazine 25-6 - 41
Instrumentation & Measurement Magazine 25-6 - 42
Instrumentation & Measurement Magazine 25-6 - 43
Instrumentation & Measurement Magazine 25-6 - 44
Instrumentation & Measurement Magazine 25-6 - 45
Instrumentation & Measurement Magazine 25-6 - 46
Instrumentation & Measurement Magazine 25-6 - Cover3
Instrumentation & Measurement Magazine 25-6 - Cover4
https://www.nxtbook.com/allen/iamm/26-3
https://www.nxtbook.com/allen/iamm/26-2
https://www.nxtbook.com/allen/iamm/26-1
https://www.nxtbook.com/allen/iamm/25-9
https://www.nxtbook.com/allen/iamm/25-8
https://www.nxtbook.com/allen/iamm/25-7
https://www.nxtbook.com/allen/iamm/25-6
https://www.nxtbook.com/allen/iamm/25-5
https://www.nxtbook.com/allen/iamm/25-4
https://www.nxtbook.com/allen/iamm/25-3
https://www.nxtbook.com/allen/iamm/instrumentation-measurement-magazine-25-2
https://www.nxtbook.com/allen/iamm/25-1
https://www.nxtbook.com/allen/iamm/24-9
https://www.nxtbook.com/allen/iamm/24-7
https://www.nxtbook.com/allen/iamm/24-8
https://www.nxtbook.com/allen/iamm/24-6
https://www.nxtbook.com/allen/iamm/24-5
https://www.nxtbook.com/allen/iamm/24-4
https://www.nxtbook.com/allen/iamm/24-3
https://www.nxtbook.com/allen/iamm/24-2
https://www.nxtbook.com/allen/iamm/24-1
https://www.nxtbook.com/allen/iamm/23-9
https://www.nxtbook.com/allen/iamm/23-8
https://www.nxtbook.com/allen/iamm/23-6
https://www.nxtbook.com/allen/iamm/23-5
https://www.nxtbook.com/allen/iamm/23-2
https://www.nxtbook.com/allen/iamm/23-3
https://www.nxtbook.com/allen/iamm/23-4
https://www.nxtbookmedia.com