IEEE Robotics & Automation Magazine - September 2021 - 37

Environment Modeling " section) and across time (the
" Long-Term Temporal Map Registration " section).
UAV Localization and Mapping
The aim of the UAV perception system is to collect highresolution
spatiotemporal, multispectral maps of the field.
These data are critical as they allow for mission planning
before the UGV actually enters the field, thereby optimizing
the time/location of ground-intervention procedures
without the risk of crop damage and soil compaction. The
perception pipeline requires two main competencies:
1) motion estimation and precise localization within the
field and 2) multiresolution, multispectral aerial mapping
based on the indicators needed to assess plant health.
The key challenges for on-field vision-based localization
are the homogeneous appearance
of crops and the accuracy of the
GPS which, when used alone, is not
sufficient to construct maps for
defining paths for UGV intervention.
To address this, we developed
an onboard state-estimation system
that combines data from a synchronized
VI sensor, a GPS sensor, the
UAV inertial measurement unit
(IMU), and, optionally, a laser altimeter
to estimate the six degree-offreedom
(6-DoF) pose. Figure 7(a)
provides an overview of the system.
The robust visual inertial odometry
(ROVIO) [25] framework is used to
produce a 6-DoF pose output based
on raw images and the IMU data
from the VI sensor. The multisensor
fusion framework [26] then combines
the ROVIO output and the
UAV IMU data to obtain state estimates
that are passed to our model
predictive controller for trajectory
tracking. To improve its accuracy and
robustness, we integrated our system
with MAPLAB [27], a framework
that has map-maintenance and processing
capabilities. On-field results
using the AscTec NEO and DJI Ma -
trice 100 UAV platforms demonstrate
a high state-estimation accuracy
compared to the ground truth from a
Leica Geosystems Total Station [see
Figure 7(b)].
High-resolution field map models
are a key prerequisite for enabling
robots in precision agriculture. To
this end, we developed a UAV environmental
modeling framework
using the pose estimate from our
localization system as well as color and multispectral camera
information over multiple flights to create spatiotemporal
spectral field models. Figure 8 depicts our pipeline.
Taking raw RGB and multispectral images and UAV poses
as inputs, we radiometrically corrected the spectral data
to create a spatial field model in the form of a dense point
cloud. For each point, the spectral reflectances in the multispectral
wavelength bands were estimated and stored.
The field evolution over time can be viewed through layered
orthomosaics generated from this data through a
custom browser-based visualization module, as shown in
Figure 9(a) and (b). We used higher-quality RGB camera
images, which recovered high-resolution field geometry,
and also used the relative position and orientation between
the RGB and multispectral camera to estimate its spectral
Ximea
RealSense ZR300
Multispectral
Camera
Color
Images
Fish-Eye
Images +
IMU
Time
Sync
GPS
Raw
ROVIO (D2.1)
UAV
Pose
Structure From Motion
(Pix4D)
Images
Radio CC Toolbox
RGB Point Cloud
+ Corrected
Camera Poses
Calibrated
Data Cube
Reprojection
Module
Camera
Extrinsics
3D Spectral Point
Cloud
Figure 8. A block diagram of our environment modeling framework depicting the sensor
suite and major software components. CC: camera calibration. (Images courtesy of Intel,
Pix4D SA, and Ximea.)
SEPTEMBER 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
37
IEEE Robotics & Automation Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2021
