IEEE Geoscience and Remote Sensing Magazine - September 2019 - 41

community-based rotation averaging while estimating
camera centers in an incremental manner [98]. To achieve
city-scale sparse reconstruction, Zhu et al. [99] grouped
cameras, performed local incremental SfM in each cluster, and then conducted global averaging among clusters.
The hybrid SfM method possesses both robustness from its
incremental aspect and efficiency from its global feature.
However, repeated BA is still needed in the estimation of
camera centers, an area that requires additional research.
Recently, semantic information was integrated into sparse
reconstruction [100]. This approach considers the semantic
SfM as a maximum-likelihood problem to jointly estimate
the semantic information (e.g., object classes) and recover
the geometry of the scene (camera pose, objects, and points).
However, because of its large memory and computational
cost, this method is often limited to small scenes and low resolution. Semantic information can also be used to constrain
feature matching and BA by semantic consistency [101].
IMAGE ORIENTATION
In SfM, camera poses are often estimated from feature correspondences by solving the perspective-n-point problem
and then optimizing by BA. Moreover, external orientation
sensors can be adopted for camera pose estimation. If UAVs
are equipped with high-quality GPS/IMUs, the camera positions and orientations can be directly estimated from the
GPS/IMU data without the need of GCPs, namely, through
direct sensor orientation or direct georeferencing [102].
Besides, orientation parameters from the GPS/IMU can
be used to initialize the camera poses and then integrate
them into aerial triangulation for BA, i.e., integrated sensor
orientation. However, UAVs are often mounted with lowaccuracy navigation sensors because of payload limitations
and the high cost of low-weight, highly precise navigation
systems. Therefore, GCPs are adopted for precise aerial
triangulation (indirect sensor orientation), which is time
consuming and laborious.

The existing SfM approaches generally rely heavily on
accurate feature matching. Some failure may be caused by
low/no texture, stereo ambiguities, and occlusions, which are
common in natural scenes. Thus, to break through these limitations, deep models have recently been applied for camera
pose estimation or localization [103]. In [104], a PoseNet was
designed to regress the camera pose from a single image in an
end-to-end manner. The traditional SfM was also modeled
by learning the monocular depth and ego-motion in a coupled way, which could handle dynamic objects by learning
an explainability mask [105], [106]. However, the accuracy of
these methods is far from that of traditional SfM. Moreover,
they are dependent on the data set, and it is difficult for them
to provide good generalization. Thus, it would be beneficial
to build more diverse data sets and encode more geometric
constraints into deep models.
STRUCTURE FROM MOTION FOR
ROLLING-SHUTTER CAMERAS
Most off-the-shelf cameras are equipped with a rolling
shutter because of the low manufacturing cost. However,
its row-wise exposure delay produces some problems. In
the presence of camera motion, each row is captured in
turn and thus with a different pose, which causes severe
geometric artifacts in the recorded image (e.g., skew and
curvature distortions). This is a considerable problem for
classical global-shutter geometric models and results in
severe errors in 3D reconstruction. Thus, new methods
adapted to rolling-shutter cameras are strongly desired.
Some works have contributed to correcting rolling-shutter distortions [107]. One technique is to use interframe
correspondences to estimate the camera trajectory and register frames. The continuity and smoothness of the camera motion between video frames can also be combined
to improve performance. Another way is to implement
correction as an optimization problem based on straightness, angle, and length constraints on the detected curves

TABLE 4. A COMPARISON OF THREE SFM PARADIGMS.*
ITEM

INCREMENTAL

GLOBAL

HIERARCHICAL

Match graph
initialization

Initialized by selected seed image pairs

All images treated equally

Atomic models

Camera
registration

Perspective-n-Point, 2D-3D
correspondences

Rotation and translation averaging

3D-3D fusion

Iterative, many times

One time

BA when merging

Advantages

Robust, high accuracy, good completeness
of the reconstructed scene

Evenly distributed errors, high efficiency

Fewer BA steps

Disadvantages

Prone to drifting errors, low efficiency

Prone to noisy pairwise matches, relatively
low accuracy, low completeness of the
reconstructed scene

Model merging, graph partition

Tools

Bundler, OpenMVG, VSFM, MVE, ColMap

OpenMVG, 1DSfM, DISCO, Theia

Research papers

* Tianwei Shen, Jinglu Wang, Tian Fang, and Long Quan, "Tutorial: Large-Scale 3D Reconstruction From Images," Asian Conference on Computer
Vision, 2016. OpenMVG: open multiple view geometry; VSfM: visual SfM; MVE: multiview environment; 1DSfM: 1D SfM; DISCO: discrete-continuous
optimization for SfM.

SEPTEMBER 2019

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

IEEE Geoscience and Remote Sensing Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2019