IEEE Consumer Electronics Magazine - March 2018 - 68

3-D location in the real world because only the location of
valid gestures is of concern. Once a valid gesture is found,
stereo matching and coordinates calculation of the hand's
mass center will follow.
Stereo matching is done to find corresponding points in a
pair of images caught by binocular cameras. Because of the
difference in camera shooting locations, one point has different locations in the right and left images. We implement stereo matching based on region similarity (seen in Figure 5).
To find the corresponding point of L (x, y) in the left image
[Figure 5(a)], a small window centered on L (x, y) is determined, and its gray-scale histogram is calculated as its feature. Search in the right image [Figure 5(b)] with the window
to find the region with largest similarity based on histogram information.
The premise of realizing stereo matching for image pairs
is that two cameras are at the same horizontal level and their
optic axes are parallel. Then, an object can be targeted in the
real world based on the matching point and its locations in
the left and right images. For a point P (x c, y c, z c) in the real
world, it corresponds to Pl (x l, y l) in the left image and
Pr (x r, y r) in the right image [seen in Figure 6(a) and (b)].
They follow the relationship in (9), where B is the distance
between the optical centers of two cameras and f indicates
their identical focal length:

Z
] xc =
]]
[ yc =
]
] zc =
\

B ·xl
xl - xr
B ·yl
.
xl - xr
B· f
xl - xr

(9)

CHARACTER RECORDING
We define two gestures in the "Gesture Recognition" section,
and since valid gestures are the key to character trajectory,
only valid gestures are processed with stereo matching and
coordinate calculation. If a valid gesture is found, we need to
judge whether the strokes come to an end. We consider continuous detection of invalid gestures as a sign of a stroke's
end. Then all calculated coordinates in a stroke are connected
to formulate the written characters. By mapping them into a
normalized plane, these characters can be recognized by the
optical character recognition (OCR) technique offered by
Hanwang Technology, China, which is useful in Chinese
character recognition. Characters written by users with their
natural hand can now be input successfully into a computer.
The framework of the whole algorithm is shown in Figure 7.

EXPERIMENT AND EVALUATION
We took 200 gesture samples to train the neural network, including 100 valid gestures and 100 invalid gestures. To increase the

Left
Image

Hand
Segment

R (x-d,y )

L (x,y )

Feature
Extraction

d
(a)

Gesture Invalid
Recognition

(b)

End
Detection

Valid
FIGURE 5. The stereo matching based on region similarity. (a) L (x, y)
is the point to be matched, and (b) R (x - d, y) is its corresponding point.

Right
Image

Stereo
Matching

Coordinate
Calculation

Trajectory
Recording

FIGURE 7. The framework of the whole algorithm.
Optical Axis of
Left Camera

Optical Axis of
Right Camera

P = (xc, yc, zc)
Pl
Ol

Y

Y

xl
(a)

Pr
xr

Base Line B

(a)

Or
(b)

FIGURE 6. A map of locations of the real-world system and the
image system. O l and O r are the optic centers of the camera lenses. P is the coordinate in the real world, corresponding to (a) Pl
and (b) Pr .
68 IEEE Consumer Electronics Magazine

^

MARCH 2018

(b)
FIGURE 8. Training samples showing (a) valid gesture samples and
(b) invalid gesture samples.
Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - March 2018
