IEEE Geoscience and Remote Sensing Magazine - September 2015 - 87

the most famous machine learning competitions such as
the Netflix Prize [5, 6], were won using ensemble methods.
As a generically applicable machine learning toolkit ensemble methods also found their way in the remote sensing
community. There are several reasons why ensemble methods are highly beneficial in remote sensing applications.
1) Ensemble methods can provide a strong increase in prediction accuracy compared to some baseline methods
(e.g., logistic regression that is often used as baseline
in several domains of machine learning classification,
even though it is less commonly applied for remote sensing applications). At the same time they typically offer
similar accuracy (arguably even a bit higher on average
across numerous real-world datasets [7]) to other state of
the art methods such as SVMs.
2) In many remote sensing applications the training set
might not be perfectly representative for the situations
in which the classifier is applied later. This holds especially for scenarios with a high dimensionality and large
data size. Ensemble methods typically show higher degree of generalizability in such situations.
3) It is often unfeasible to learn one single strong classifier, e.g. a single SVM, if the training dataset is too large.
Ensemble methods have better big-O complexity and
inherently support parallelism by training a multitude
of weak learners, each of them on a small portion of the
training set. This allows running the training process on
multiple cores or computers which is highly attractive in
many situations.
Formally, an ensemble is a technique for combining
numerous weak learners in an attempt to produce a strong
learner. An ensemble is a supervised learning method,
since it has the capacity to be trained and then used to perform predictions. As such, the ensemble also represents a
single hypothesis in the solution space. However, this hypothesis is not necessarily contained within the space of
the models which were used to construct the ensemble.
Therefore ensembles typically have more flexibility in the
functions they can represent, which can result in a reduction of model bias [8]. Considering the typical bias-variance decomposition and the bias-variance tradeoff [9],
an increase in model complexity is often associated with
an increase in variance, since the more complex model is
potentially more prone to overfitting the training data.
This effect is encountered to a different extent in various
ensemble methods, but some of them are specifically designed to reduce the variance part component of the error
(e.g., Bagging).
Although numerous variations of tree-based ensembles have been proposed in the literature, three dominant
algorithms have emerged from successful application
in a variety of application areas. The first of them and
perhaps the most well known is the Random Forest (RF)
algorithm, as proposed by Breiman [10] which recently
was also applied to hyperspectral imagery [11], [9]. The
second one is the Extra Trees (ET) algorithm, that pushes
september 2015

ieee Geoscience and remote sensing magazine

the randomization idea further, targeting an even more
significant reduction in variance compared to the Random Forest. Finally, the third algorithm, often called Gradient Boosted Regression Trees (GBRT) or Gradient Tree
Boosting is of completely different nature, as it aims to
reduce mostly model bias. In this paper we are comparing the goals, assumptions and limitations of the three
algorithms under a unified framework of bias-variance
decomposition. Moreover, we discuss their advantages
and disadvantages, both in general as well as in particular
for the field of data fusion in remote sensing. We compare
their performance in challenging remote sensing datasets
with different parameters, and we conclude with practical
considerations for the selection of the appropriate algorithm given the data at hand.
It should be noted that the current paper focuses on
comparing the tree-based algorithms that are typically
available in most machine learning packages (e.g., implementations of these algorithms are freely available
in at least Matlab, Python,
and R among others) and
AN ENSEMBLE cOMBINES
thus can be easily integrated
NuMEROuS wEAk
in a practitioner's toolbox.
LEARNERS TO pROducE A
More recent variants like for
STRONG LEARNER, wHOSE
example the Rotation Forest
HYpOTHESIS IS NOT
that have shown very promNEcESSARILY cONTAINEd IN
ising results in the field of
THE SpAcE OF THE MOdELS
remote sensing [12], [13],
uSEd TO cONSTRucT THE
and [14] but are still not easily available (e.g., there are
ENSEMBLE.
fewer open source implementations) are not part of
the current study. Excluding
these recent works, the selected algorithms that we study
in the paper can be considered as a generalization of most
tree-based ensembles, for example Tree Bagging is a special
case of Random Forest (when in every node the full feature set is considered for doing the split [10]) while most of
the boosting algorithms can also be derived from Gradient
Boosted Regression Trees (e.g., AdaBoost is derived from
GBRT using a special cost function, as described in [15]).
The rest of the paper is structured as follows: Section 2
gives an overview of the state of the art in ensemble learning, both inside and outside the field of remote sensing.
Section 3 provides the necessary theoretical foundations
of ensemble learning while Section 4 provides a detailed
summary of the three most widely used ensemble learning
techniques including best practices. Experimental results
on three remote sensing datasets are shown in Section 5
before the paper ends with a conclusion in Section 6.
2. A SHORT HISTORY OF ENSEMBLE LEARNING
Combining the results of several classifiers in a systematic
way has been a significant topic of research since the late
1980s and early 1990s. It was the work by Breiman [16],
87

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2015