IEEE Geoscience and Remote Sensing Magazine - September 2019 - 42

most common photoconsistency measures include normalized cross correlation, the sum of absolute or squared
differences, mutual information, census, rank, dense feature descriptors, gradient-based algorithms, and bidirectional reflectance distribution functions [110]. MVS is often formulated as a function of illumination, geometry,
viewpoints, and materials and thus can be regarded as a
constrained optimization problem solved by convex optimization, Markov random fields, dynamic programming,
or the graph-cut or maximum-flow methods [28].
Most conventional MVS matching techniques are adapted directly for UAV image-based surface reconstruction
[111]. Considering the perspective distortions in oblique
images, epipolar rectification is performed based on the
cost of angle deformation before MVS matching [112]. To
minimize the influence of boundaries, a hierarchical and
adaptive phase correlation is adopted to estimate the disparity of the UAV stereo images [113]. In addition, some
tricks have been proposed to improve the performance of
conventional methods, including graph networks, image
grouping, and self-adaptive patches [69].

to estimate the camera motion and thus rectify the rollingshutter effect. This method is sensitive to feature choice and
extraction. Recently, CNNs were adopted to automatically
learn the interplay between scene features and the row-wise
camera motion and correct the distortions [108]. Largescale data sets are obviously required. They often train
CNNs on synthetic data sets, which may be different from
real cases, but this is a promising approach.
In the case of conventional SfM, rolling-shutter effects
are modeled [37], [109]. This complex model is shattered
into a constellation of simple global-shutter, linear-perspective feature cameras. The poses (i.e., rotation and translation) of each feature are linearly interpolated according to
their vertical position in the image between successive key
poses. Usually, a linear interpolation is used for translation
and a spherical linear interpolation for rotation. In general,
one may insert as many key poses as tracked features.
CHALLENGES IN AERIAL TRIANGULATION
Although aerial triangulation/SfM is a long-standing problem, it still faces many hurdles, such as very large-scale and
high-efficiency SfM, AAT with arbitrary images, and multisource data AAT (ground/street images and UAV images).
Additionally, there is a long way to go with semantic SfM
and deep CNNs for camera pose estimation.

LEARNING-BASED MULTIVIEW STEREO
The aforementioned methods use handcrafted similarity
metrics and engineered regularizations to compute dense
matching and are easily affected by sudden changes in
brightness and parallax, repeated/no textures, occlusion,
large deformations, and the like.
Recent success in deep learning research has attracted
interest in improving dense reconstruction. Numerous
works apply CNNs to learn pairwise matching cost [114]
and cost regularization [115] and to perform end-to-end
disparity learning [116]. However, most methods focus on
stereo matching tasks, and it is nontrivial to extend them
to multiview scenarios. Furthermore, the extended operations do not fully utilize the multiview information and
lead to less accurate results. Input images could also be of
arbitrary camera geometries.
There are fewer works on learned MVS approaches. SurfaceNet [117] and Learned Stereo Machines [118] encode
camera information in the network to form the cost volume
and use 3D CNN to infer the surface voxels. However, these

DENSE RECONSTRUCTION
A complete workflow of 3D construction includes SfM,
dense reconstruction, surface reconstruction, and texture
mapping [15], as shown in Figure 11. Once a set of UAV
images is oriented (the known camera poses), the scene can
be densely reconstructed by dense image matching-i.e.,
multiview stereo (MVS) matching-which is the focus of
this section.
MULTIVIEW STEREO RECONSTRUCTION
Numerous MVS algorithms have been proposed, e.g.,
semiglobal matching, patch-based methods, and visibilityconsistent dense matching [28]. To search for correspondences, similarity or photoconsistency measures are often
adopted to compare and estimate the likelihood of two
pixels (or groups of pixels) being in correspondence. The

(a)

(b)

(c)

(d)

(e)

FIGURE 11. The steps in image-based, multiview 3D reconstruction. Based on (a) UAV images, (b) SfM is performed to estimate camera poses and sparse 3D structure. (c) Dense reconstruction (MVS matching) is then adopted to generate the dense 3D scene structure. (d) Surface
reconstruction (mesh generation) is conducted to generate a surface model. (e) After texture mapping, the real 3D model is reconstructed.

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

SEPTEMBER 2019

IEEE Geoscience and Remote Sensing Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2019