Instrumentation & Measurement Magazine 26-2 - 22

block, we introduce HWA and DTN, as described in the next
sections.
Table 1 lists the detailed architecture setting, where: Ni
is the number of blocks in each stage; C is the Channel dimension;
heads is the number of heads in hybrid window
attention; and window size is the region size in hybrid window
attention.
Hybrid Window Attention
Hybrid window attention is proposed to add cross-window
connection based on the introduction of pooling attention
and window attention. It calculates all the local attention
within a window, and by this way, the input features contain
global information [25]. Specifically, in order to process
the 2D image, we reshape X∊RHxWxD
tened 2D patches XR
NP2 D



into a sequence of flat,
where (H,W) is the resolution
of the original input image.
For an input sequence X∊RLxD
,WK,WV∊RLxD
, (where L=P2
C, denotes the
original image resolution; (P,P) is the resolution of each image
patch; and D denotes the number of image channels) a linear
projection WQ
is applied, followed by a pooling
operation (P) for the query, key and value tensor:
Q P XW K P XW V P XWV
Q
Q
,
 K
K
,
 V
where the length  L of QR 


lengths of K and V can be reduced by PK


andPV
self-attention can be expressed as:
, , max(
and PV
Z Attention Q K V Soft QK D V
T
/
)
to compute a length-flexible output sequence Z∊RLxD
downsampling factors PK
different from those applied to the query sequence PQ
    
LD can be reduced by PQ
(1)
and the
. Then, pooled
(2)
. The
for key and value may be
. The resolution
between different stages of DHT can be reduced by
pooling query Q and will significantly reduce the computational
complexity by pooling key K and value V. The relevant
analysis is as follows, in which we present the complexity of
the model.
We assume that the input and output features have the
same size H×W×C. Hybrid window attention will be calculated
in the local window for self-attention. There is
segmentation of images by non-overlapping windows. Each
window contains P×P patches. To ensure that the image (H×W)
is divisible by the window (P×P), when calculating attention,
we use the padding method.
 R H
h

T
C

x 


x Concate  



Concate  
hH

hH  
h


 
2
h
where γ,β are two learning rate parameters of affine transformation
in the model. H represents the number of model
heads, and
, 
2
h

T
 R
C
H is the normalization constant
of DTN. The 'Concate' in the equation means to
concatenate together the normalization constant of DTN.
This design is accomplished in two steps. Firstly, patches
with different attention distances are attended to by the attention
heads, and different self-attention heads are encouraged
to conduct different background modelling. Secondly, with
two different statistics (inter -token and intra -token), we design
the DTN by calculating the normalization constant for
each head of the hybrid window attention.
Dataset Description and Experiment
Configuration
In our experiments, the NEU surface defect dataset [26] and
the DAGM dataset [27] are utilized to verify the validity of our
proposed approach. First, we describe the two datasets used in
this paper and then show the experimental setup.
Dataset
Stage
1
2
3
4
22
Table 1 - Detailed architecture specifications
Output Size
DHT
H/4×W/4×C
H/8×W/8×C
H/16×W/16×C
H/32×W/32×C
N1:4; C=96; heads:3; Window size:7
N2:4; C=192; heads:6; Window size:7
N3:12; C=384; heads:12; Window size:7
N4:4; C=768; heads:24; Window size:7
IEEE Instrumentation & Measurement Magazine
The NEU surface defect dataset contains six types of defects
(Fig. 2): rolled-in scale (RS), patches (Pa), crazing (Cr), pitted
surface (PS), inclusion (In) and scratches (Sc), with 300 images
of each defect and an image size of 200×200 pixels. In the classification
task, we randomly
select 70% of the data as
the training set and the remaining
30% as the testing
set [28].
The DAGM dataset was
generated manually and
contains six classes of image
samples of 512×512 pixels
April 2023

(4)
In this model, the computational complexity of our hybrid
window attention is:
 
 2 22

=HGWG=HW/r2
HWA HWN C HWC N CSS (3)
22
where C denotes the Channel dimension, h and w represent
the height and width of the image, respectively, and
NS
is the number of sampling points. It can be
concluded that the computational cost has a linear complexity.
Channel size is insignificant compared to the cost of attention
calculation.
Dynamic Token Normalization (DTN)
This article defines DTN with a unified description. The normalization
constant of DTN is obtained by computing the
token's statistics. We reshape the image into a sequence of tokens
x∊RLxD
, and DTN normalizes the token by the following:
Instrumentation & Measurement Magazine 26-2

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 26-2
