IEEE Robotics & Automation Magazine - December 2015 - 154

Fifth Edition
The fifth edition [20] encouraged participants to use 3-D
information (point cloud files) with the inclusion of rooms
completely imaged in dark. It also introduced the
identification of the objects present in the scene, as well as a
proposed classification method similar to that of the 2012
edition, used to present a baseline score of 5,004.75 out of
7,030 points. The highest result, 6,033.5 points, was obtained
by the Institute of Computing Technology, Chinese Academy
of Sciences, Beijing, China (MIAR ICT) [50]. They proposed
the use of kernel descriptors [51] for both visual and depth
information, while PCA was applied for dimensionality
reduction. They used SVM classifiers and managed object
recognition and room
classification separately.
Actually, both problems
We plan to manage
were expected to be
handled together, but
both room classification
none of the participants
presented a proposal
and object recognition
where the appearance of
the object (or lack thereof)
problems jointly.
is used to classify the
room. The group from the
University of Sfax National School of Engineers, Tunisia
(REGIM) [52] achieved a score of 4,638.25 points, slightly
lower than the baseline score. Visual images were processed
using Pyramid Histogram of Visual Words [53], and linear
SVMs were selected as the classification model.
As a general remark, we can point out that most editions
were won by those participants taking advantage of the introduced novelties. Namely, those proposals that ranked first in
the second and third editions were based on the spatial geometry acquired from stereo images. The winner of the fourth
edition was the only proposal using range information, and a
similar scenario was found in the 2013 edition.
Analysis of the Results
In addition to the generation of solutions to the problem
provided by each edition, the Robot Vision task has served
for sharing techniques and knowledge between worldwide
researchers. This experience has helped several robotic
laboratories generate their own place classifiers and also
develop novel approaches that have been successfully deployed in different environments. Here, we review some of
these proposals.
The University of Glasgow only participated in the first
edition of the challenge ranking second for the optional
task [54], but their novel matching approach was used to
develop a robust localization system [55]. The PicSOM
content-based image retrieval system [56] from the University of Helsinki was evaluated in the second edition of the
challenge. Despite their low ranking (fifth [57]), the experience gathered from the Robot Vision challenge allowed
them to develop a concept detection system relying on the
use of kernel maps [58]. Members from the CVG group,
154

IEEE ROBOTICS & AUTOMATION MAGAZINE

DECEMBER 2015

winner of the second and third editions, also presented an
extension of their proposals, namely, the visual mapping
and localization problem [59], for outdoor environments.
Another remarkable proposal drawn from the Robot Vision competition is the generalist use of Fisher Vectors for
image classification, as presented by the winner of the
fourth edition of the task in [60].
The use of kernel descriptors, as part of the MIAR ICT
participation in the 2013 edition, can be considered the
most outstanding technique presented in the Robot Vision
challenge [50]. This technique generates rich path-level
features from pixel attributes, which are then used to
generate image representations suitable for further
classifications. Kernel descriptors have been previously used
for scene labeling and classification [61], but the temporal
continuity of the sequences of images has not been exploited
in previous proposals. The MIAR ICT team approach
smooths the score of their algorithm using a smoothing
window, which increases its accuracy by 8.43% when
validated against the validation sequence.
Conclusions
In this article we presented an overview of the five consecutive editions of the Robot Vision challenge at ImageCLEF.
First, we described the challenge and why we consider it a
relevant and important competition for the robotics science
community in particular, and to the research community in
computer vision in general. We also described some related
competitions and how the tasks proposed in each challenge
have evolved from the first edition to the last one. The challenge has evolved over the different editions: different information sources, different scene categories, visual and
depth images, intrascene object information, the amount of
training and validation data, and baseline useful software
have been used across editions. As a result of these changes,
different data sets have been released and publicly provided
as a testbed for the visual scene classification problem. Finally, we presented the more relevant proposals as well as a
summary of the best results obtained in the five editions of
the competition.
As a main conclusion, we argue that the Robot Vision
challenge has provided valuable resources including data sets,
benchmarking techniques, and state-of-the-art solutions to
the visual place classification problem. Moreover, the challenge has contributed to the generation of a semantic localization researcher community.
For future editions of the challenge, we plan to manage
both room classification and object recognition problems
jointly. Subsequently, we will challenge participants to classify
rooms using the list of objects recognized in a scene as inputs.
Also, the use of online development environments will be
considered.
Acknowledgments
This work was supported by grant DPI2013-40534-R of the
Ministerio de Economia y Competitividad of the Spanish

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2015