IEEE Consumer Electronics Magazine - March 2018 - 69

flexibility of the network, gestures were sampled in different
directions (seen in Figure 8). The camera used in the experiment
is an RoHS 1024 × 768 Color 3.8 mm Bumblebee2. To evaluate
the performance of our system, we wrote three groups of characters in the air using the gestures we have defined. Each group of
characters is the same character written in three different planes
(seen in Figure 9). Plane S1 is parallel to the imaging plane of
the stereo camera. S2 rotates to the left and has an angle of 45°
compared with S1. S3 rotates to the right and also has an angle
of 45° compared with S1, as seen in Figure 9. We chose these
different writing planes to test our system's flexibility to deal
with various types of free input, which means there are no
restrictions that a user must write the character strictly facing the
camera. Tracking gap m in our experiment is 5.

S3
45°
45°

GESTURE-RECOGNITION EXPERIMENT AND
EVALUATION

CHARACTER-RECORDING EXPERIMENT
AND EVALUATION
With stereo cameras as the input device, the 3-D location of the
targets can be conveniently obtained. Combined with the defined
writing gestures, we can obtain the 3-D trajectory of a user's
hand. For a valid gesture, we calculate its coordinates in the real
world using the techniques explained in the "Character Recording" section and connect them to formulate the written characters. Figures 11-13 show the recording trajectories of some of
our input characters. As each sample character is written in three
different planes, all characters are recorded three times, corresponding to different planes. We can see that the characters are
recorded correctly regardless of the writing planes. This shows
that our algorithm successfully records 3-D characters. After
these characters are normalized, they are recognized correctly by
Hanwang OCR v8 (http://ka.hanwang.com.cn/en/). This shows
that free 3-D input is successfully realized.

Stereo
Camera

FIGURE 9. The writing planes in the experiment. Plane S1 is parallel to the imaging plane of the stereo camera. S2 rotates to the
left and has an angle of 45° compared with S1. S3 rotates to the
right and also has an angle of 45° compared with S1.

Accuracy of Gesture Recognition

In our experiment, every gesture checked is labeled as valid
or invalid by the trained network. First, to evaluate the performance of the Hu moments, we carried out an experiment to
find Hu moments' effect on gesture recognition. As seen in
Figure 10, the number of Hu moments represents how many
features we used beginning with h1; e.g., number 1 means we
use only h1, and number 3 means we use h1, h2, and h3. We
can see that the accuracy of gesture recognition increases
with the number of features used, and the increasing speed is
even more rapid when the number is small.
To evaluate the performance of our gesture recognition,
we count the total number of gestures that are checked and
the number of ones that are right-labeled. As seen in Table 1,
we have statistics for all three writing planes. The accuracy
in three different directions is equally high. One reason is
that Hu moments describe the gestures in a robust way. In
addition, the strategy of sample selection increases the network's ability to adjust to shape change. This shows that our
gesture recognition is able to deal with free input in different
directions, and this greatly increases convenience when a
user interacts with computers in a VR environment.

Accuracy of Different Hu Moments

100
90
80
70
60
50
40

3
4
5
Number of Hu Moments

FIGURE 10. The accuracy of gesture recognition depends on different numbers of Hu moments.

Table 1. The accuracy of gesture recognition.
Planes

Sum

num(checked)

1,232

1,084

1,165

3,481

num(right-labeled)

1,162

1,013

1,085

3,228

recognition accuracy

0.94

0.93

0.936

Note: num(checked) is the number of all gestures that are checked
in our experiment and num(right-labeled) is the number of right
recognized gestures, including valid and invalid gestures. Accuracy is
the percentage of right-labeled gestures compared with the number
of checked gestures.

MARCH 2018

IEEE Consumer Electronics Magazine

http://ka.hanwang.com.cn/en/

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - March 2018