IEEE Geoscience and Remote Sensing Magazine - September 2019 - 44

In [132], images are corrected by global transformations
derived from the relationships between GCPs and the corresponding image points. Considering the exterior orientation inaccuracy of the GPS/IMU and the difficulties in acquisition of GCPs, another orthorectification technique is
based on registration with the aerial/satellite orthorectified
map [130]. This approach is more efficient and convenient
because it avoids complex aerial triangulation and DEM
generation and the laborious acquisition of GCPs. But its
mandatory prerequisite is the reference maps.
TRANSFORMATION-BASED IMAGE STITCHING
Orthorectification-based image stitching can rectify geometric distortions and provide geographic coordinate information, but it is generally computationally complex and
time consuming, which makes it unsuitable for time-critical remote sensing applications [133] such as disaster, emergency, and security monitoring. The transformation-based
technique, however, provides an effective mosaic method
based on transformations calculated from matching correspondences between adjacent images [134].
A simple approach is to exploit one global transformation to align images [135]. However, it works well only under the assumptions of roughly planar scenes or parallaxfree camera motion [66], which may be violated in most
UAV-based data acquisition cases. Although advanced image composition can mitigate the stitching artifacts generated by these methods, they remain when there are misalignments or parallax.
To deal with this problem, spatially varying warping
methods have been proposed for image alignment. One is
to adopt multiple local transformations to locally align images, including as-projective-as-possible warping [136] and
the elastic local alignment model [137]. The other is to consider registration as an energy optimization problem, with
geometric or radiometric constraints based on the mesh
optimization model [131], [138]. Local transformations can
also be integrated with mesh models to provide good stitching [139]. Spatially varying warping models can handle
moderate parallax and provide satisfactory stitching performance, but they often introduce projective distortions,
e.g., perspective and structural deformations, because of the
nonlinear nature of these transformations. Some methods
have been proposed to handle distortions, such as the global similarity prior model [140] and structural constraint
model [139], but more effort needs to be invested in stitching images accurately with reduced distortion.
Another approach is seam-guided image stitching [141],
which has the potential to handle large parallax. Multiple
transformation hypotheses can be estimated from different groups of feature correspondences. Seam-line quality is
then utilized to evaluate the alignment performance of different hypotheses and select the optimal transformation.
This method adopts a local transformation for global alignment, so it would become trapped when tackling images
with complex multiplane scenes.
44

CHALLENGES IN IMAGE STITCHING
Although numerous stitching methods have been developed, there are open problems, especially in stitching images with efficiency, registration accuracy, and reduced distortion. More research should be devoted to high-efficiency/
real-time image stitching, large-parallax image stitching,
and distortion handling. Additionally, there have recently
been some attempts using deep learning in homography estimation and image dodging [142], [143]. However, there is
still much room for improvement. This is a promising and
worthwhile direction for research.
MULTISENSOR DATA REGISTRATION
With the advent of increasingly available sensors, UAV-RS
platforms are often equipped with multiple tools (e.g., visible cameras, infrared sensors, or laser scanners) that can
either collect a variety of data at a time to achieve multiple
tasks or integrate these complementary and redundant data
for better understanding of the entire scene. However, the
data from multiple sensors often have dramatically different characteristics, e.g., in resolution, intensity, geometry,
and even data dimension, due to different imaging principles. This poses a huge problem for integrating multisensor
data for remote sensing applications [144].
Multisensor data registration is a mandatory prerequisite. The data are then fused for interpretation. Because of
space limitations, this section focuses on multisensor data
registration. Remote sensing data fusion is not discussed
here but can be explored in the surveys in [145] and [146].
MULTIBAND IMAGE REGISTRATION
The registration of multiband images, e.g., visible and
infrared images or visible and synthetic aperture radar
images, has caused great concern in recent years. The
area-based method commonly adopts intensity statistics
information, such as mutual-information and entropybased measures [147], to handle the large appearance differences. These techniques have difficulty handling large
radiometric distortions because they are mainly based on
image intensities. But structure features, such as gradients, edge information, local self-similarities, and phase
congruency, are more robust to radiometric changes and
are integrated as similarity metrics to improve registration
performance [148]. However, these methods are computationally expensive.
Feature-based registration often extracts geometric features and then matches them based on descriptor matching [149], [150]. However, traditional gradient- or intensitybased feature descriptors are not suitable for multimodal
image matching because of the large gradient differences.
Thus, some structure features, e.g., line segments and edges, are described by geometrical relationships, edge histograms, or log-Gabor filters [151]. Figure 13 shows some
promising results and demonstrates the effectiveness of
description based on structure information, but the performance is far from satisfactory. Therefore, much room
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

SEPTEMBER 2019

IEEE Geoscience and Remote Sensing Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2019