The Bridge - Issue 3, 2021 - 20

Feature
Analyzing Service Mesh Performance
with the guarantee afforded through the act of dryrunning
the service mesh and application configuration
before production deployment. The Meshery and Service
Mesh Performance at GitHub Action (https://github.
com/layer5io/meshery-smp-action) offers the ability to
adaptively analyze application performance as a gate in
your continuous delivery pipeline. In this way, the SMP
specification facilitates a measurement index that can be
referenced when rolling out new versions of a service
with this advanced canary technique.
Through Meshery, techniques to mirror non-idempotent
requests allow replay of user requests without fear of
impacting the current version of your application. Through
the use of intelligent network functions, embedded in
WebAssembly (WASM) programs, to facilitate real user
request reenactments, the most value can be extracted
out of your pipeline. For example, repeatability of test
scenarios can be investigated using performance profiles
and cloud native orchestration and results can be
baselined and compared.
B. Analyzing Performance Measurements
We have often seen inefficiencies in the ratio of resource
usage versus resources applied. Since the mesh elements
i.e. the ingress and sidecars share resources with one or
more of the application containers, there may be more
resources left to be utilized. Tail latencies decrease with
the increase in number of cores for all 1, 10 and 100
clones but increase with the increase in the number
of connections. Data for various connection counts, as
shown, indicates that performance degradation with Istio
(https://istio.io) shows up with input RPS more than
1000. In a top-down microarchitectural analysis (TMA)
[10], the front proxy is pinned to a single core, the sidecar
+ flask app is pinned to another core, and the number of
microservices are scaled up. It is observed that
(cf. Figure 2):
* Frontend Bound% decreases with increase in number
of microservices and Core Bound % increases.
* Memory Bound % increases with increase in the
number of microservices.
* L1 and L3 Bound% decreases for both the service cores
on which the front -proxy is running as well as the core
where the sidecar+flask app is running with number
of microservices.
In customer environments, the size of the cluster as well
as the amount of incoming traffic will have an impact on
the number of workloads and Envoy microservices. The
underlying hardware and L4 networking on each node in
the cluster will also impact the performance observed.
A call stack and cycles spent analysis of a deployment
with 1-20 sidecars on a specific 40 core system with a
10G NIC shows bottlenecks spread between:
* Envoy: TheadLocalStorage-Hashset-Match,
* Linux kernel bottleneck spread between
Libpthreadscheduling and Libevent,
* Envoy buffer slice management and TCP filter, if
message sizes or file transfer sizes increase to 1M, and
* Crypto operations when TLS is enabled.
Our initial studies show that the optimal service mesh
setup for the tolerable latencies and the best RPS
may include:
* Exclusive threads allocated to Envoy processing,
* Reduced memory contention by allocating more
memory bandwidth which can be controlled
dynamically,
* Load balancing of worker threads among the among
cores which may require less IO switching, and
* Optimized memory copies with signals incorporated in
addition to events (libevent).
A number of accelerations and offloads to SMART
NIC or other processing elements like IPUs and
DPUs are becoming available. How does the service
mesh efficiency and performance benefit from these
deployment options needs to be defined and measured.
Cycles and cores saved in the host cores versus offload
cores, which may be of different architectures and/
or performance range, needs to be quantified and
benchmarked.
C. Being Precise in Performance Studies
When measuring sub-millisecond response times,
the noise floor of the environment as well as the
sensitivity of the tooling may become dominant factors
in measurements. Noisy neighbors, scheduler fairness,
garbage collection, connection-reuse patterns, and even
specifics in the timing of requests being sent may change
noise floors. Hence, similar measurements performed
using different systems and tools may diverge an order
of magnitude in absolute terms. As a quick survey of
load generators by way of those included in Meshery, the
differences are noteworthy and justify their use under
different circumstances.
THE BRIDGE
https://github.com/layer5io/meshery-smp-action https://github.com/layer5io/meshery-smp-action https://www.istio.io https://hkn.ieee.org/
The Bridge - Issue 3, 2021

Table of Contents for the Digital Edition of The Bridge - Issue 3, 2021
