IEEE Robotics & Automation Magazine - December 2013 - 51

3-D Scene

IR Projection
Pattern
(Fine-Grained Texture)

(a)

(b)

(c)

(d)

(Left)
Projector
(Right) Camera

Figure 2. The active structured light principle behind representative
RGB-D cameras. The IR projector and camera form a stereo pair: the
projector shoots a fixed, locally unique pattern and paints the scene
with IR texture; the camera, calibrated to the projector and with the
pattern memorized, computes stereo disparity and depth for each of
the scene points. (Adapted from [4])

known, memorized pattern to substitute for one of the stereo
cameras (structured light stereo). Figure 2 shows the basics of
the structured light setup. One of the two cameras is replaced
by an IR projector. The IR pattern is generated by an IR laser
dispersed by a proprietary dual-stage diffraction grating. The
IR camera knows the locally unique projection pattern and
how the pattern shifts with distance, so a local search can
determine the shift (disparity), and depth can be computed
through triangulation. The advantage of the structured light
approach is that it tends to generate fewer false positives since
the pattern is known. On the other hand, projected texture
stereo can potentially work outdoors, taking advantage of natural as well as projected texture.
While the PrimeSense designs of the IR pattern are proprietary, the analysis of Kinect by Willow Garage [3] and
Konolige's work on projected texture stereo [4] shed light on
the design of projection patterns and the stereo algorithms
needed. Konolige showed that Hamming distance patterns
work better than De Bruijn or random patterns and can be
further optimized through simulated annealing. With projected patterns, stereo correspondence becomes easy: a standard sum-of-absolute-difference block-matching algorithm
works well enough. Figure 3 shows an example of active versus
passive stereo from [4]. Using a standard real-time stereo algorithm, the difference in depth density is dramatic. Although
passive stereo fails in most parts of the scene [Figure 3(a) and
(b)], active stereo manages to recover depth almost everywhere [Figure 3(c) and (d)]. The PrimeSense devices appear to
use a similar block-matching algorithm with 9 # 9 blocks, with
the pattern most likely optimized for local disambiguity. The
output depth is at 640 # 480 resolution and 30 frames/s, reasonably accurate between the range of 0.8-4 m. As it is a stereo
camera, the depth accuracy diminishes quadratically with distance, about 2-5 mm at 0.8-m distance and about 1-2 cm at
2 m, sufficient for many applications.

Figure 3. Stereo vision results (a), (b) with normal lighting
conditions: (a) image and (b) depth, and (c), (d) with projected
texture pattern: (c) image and (d) depth. Stereo techniques have
problems with indoor scenes where textureless areas abound, and
illumination is often poor. For a simple scenario of a mug on a table, a
standard local search algorithm fails to compute depth in most parts
of the scene (depths are color coded, with missing depth shown
as gray). In comparison, an active stereo system projects a highfrequency pattern and effectively paints the scene with texture, in
which case the same stereo algorithm succeeds in computing a dense
depth map [4].

Depth and RGB-D Calibration
A stereo setup must be calibrated internally (to eliminate lens
distortion and to find the principal points) and externally (to
find the offset between the cameras). For the PrimeSense
devices, both the projector and the IR camera exhibit low distortion. The projector is based on a laser and diffraction grating and has inherently excellent distortion characteristics.
The IR camera also exhibits low intrinsic distortion (see [3]),
probably from careful lens selection. The external correspondence between the camera and the projector is probably
determined by a factory calibration, but this is a conjecture.
In any event, a rigid mount ensures that the devices do not
have to be recalibrated during use.
Another calibration area is the relationship between the
IR and RGB cameras, to map the depth image into correspondence with the RGB image (called registration by the
OpenNI drivers). The cameras are placed close together
(about 2.5-cm offset) to reduce parallax. Again there appears
to be a factory calibration that is particular to each device.
Given a known offset between the IR and RGB images, the
IR image is mapped to a corresponding RGB point by first
converting it to 3-D coordinates, transforming from the IR
camera frame to the RGB frame using the known offset, and
then reprojecting the 3-D points to the RGB image (with
Z-buffering). This mapping can be done either on the device
(ASUS Xtion Pro) or in the PC driver (Kinect). It is worth
noting that the PrimeSense devices have the ability to synchronize the capture time of the IR and RGB images, but
time synchronization is only available on the ASUS Xtion
Pro, as it is turned off on the Kinect.
DECEMBER 2013

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

51
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2013
