IEEE Computational Intelligence Magazine - August 2021 - 31

FIGURE 5 Detailed structures of the best structure discovered on ImageNet. " SAConv2 " and " SAConv4 " denote split-attention bottleneck convolution
layer with radix of 2 and 4, respectively.
and AMDIM /smalllarge
with higher accuracy. Compared with
AMDIM ,large which is by far the best method of manual
design that we know of, FaUNAE (ProxylessNAS) achieves
not only a better performance (68.1 vs. 68.3), but the
model size was also reduced by about 20 times (626M vs.
30M). FaUNAE also performs better than the structure
sampled randomly from the search space described in 3.1
on Top-1 accuracy (68.3 vs. 66.2). When compared with
other NAS methods which use the same search space, such
as ProxylessNAS, FaUNAE obtains a better performance
with higher accuracy (67.8 vs. 68.3) and is much faster
(23.1 vs. 15.3 GPU days). In addition, we calculated the
FLOPs of different models separately. The FLOPs of
ResNet50, FaUNAE (ResNet50) and FaUNAE (ProxylessNAS)
are respectively: 4.11G, 3.87G and 4.08G. When the
FLOPs are approximately the same, FaUNAE (ProxylessNAS)
achieves better performance.
We also set different initial structures 0a including random
structure, ResNet50, and the structure searched by ProxylessNAS
on ImageNet100. As shown in Table 2, we find that the
better the initial structure, the better the performance, which
shows the importance of the prior knowledge. For the structure
(Fig. 5) obtained by FaUNAE on ImageNet, we find
that the structure on unsupervised learning prefers a small
kernel size and a split-attention convolution [35], which also
shows the effectiveness of split-attention convolution and the
rationality of FaUNAE. We searched the architecture on ImageNet
for many times, which needs further research in our
future work.
D. Results on Object Detection
and Segmentation
Learning transferable features is the main
goal of unsupervised learning. When used as
fine-tuning initialization for object detection
and segmentation, ImageNet supervised pretraining
is the most influential (e.g., [21, 22,
20]). Next, we perform experiments on the COCO [18] data
set and the PASCAL VOC [19] data set, and compared FaUNAE
with other methods.
1) Results on PASCAL VOC
We use Faster R-CNN [22] as the detector and the evaluated
network obtained on ImageNet as the backbone,
with batch normalization tuned, implemented in [48]. All
layers are fine-tuned end-to-end. We fine-tune for 18k
iterations (~18 epochs) on PASCAL VOC trainval07+12.
We evaluate the default VOC metric of AP50 on the VOC
test2007 set.
In Table 3, we can see the results on trainval07+12. FaUNAE
with ImageNet is better than ResNet50 with the same
method (MoCo v2): up to + 0.9 AP50, + 2.3 AP, and + 2.6
AP75. Also, our FaUNAE is better than the ImageNet supervised
counterparts, which shows the usefulness and effectiveness
of unsupervised learning.
TABLE 3 Results under object detection on PASCAL VOC
with Faster R-CNN.
ARCHITECTURE METHOD
RESNET50
RESNET50
RESNET50
FAUNAE
Super.
MOCO V1 [1]
MOCO V2 [11]
MOCO V2
AP50
81.3
81.5
82.4
83.3
AP
53.5
55.9
57.0
59.3
AP75
58.8
62.6
63.6
66.2
TABLE 4 Results of object detection and instance segmentation on COCO with
Mask R-CNN. APbb means bounding-box AP and APmk means mask AP.
ARCHITECTURE METHOD
RESNET50
RESNET50
FAUNAE
Super.
MOCO V1 [1]
MOCO V2
APbb
40.0
40.7
43.1
APbb
50
59.9
60.5
63.0
APbb
43.1
44.1
47.2
75
APmk
34.7
35.4
37.7
APmk
50
56.5
57.3
60.2
APmk
75
36.9
37.6
40.6
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 31
Conv 3×3
SAConv4 3×3
SAConv2 3×3
SAConv4 5×5
SAConv4 3×3
SAConv2 3×3
128×28×28
SAConv4 3×3
SAConv4 3×3
SAConv4 3×3
256×14×14
SAConv4 3×3
Conv 3×3
SAConv4 3×3
SAConv4 3×3
SAConv2 3×3
SAConv4 3×3
512×7×7
SAConv4 3×3
SAConv2 3×3
3×224×224
32×112×112
64×56×56

IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021