IEEE Robotics & Automation Magazine - September 2015 - 104

assessment of bipedal locomotion that could be applied and
shared across different research communities.
Benchmarking
In the field of humanoid robots, the main obstacle in identifying common benchmarks is that different methods and metrics
are typically employed for specific robotic systems and functional scenarios. Benchmarking of humanoid locomotion is
typically approached on a competition-based level and is mostly focused on global functional goals (e.g., playing soccer,
avoiding obstacles, and
climbing stairs [1], [3]). In
the field of wearable roThe proposed scheme
bots, performance is usually reported in terms of
can be taken as a starting
the effects on the user's
motor function. New stanpoint for a global iterative
dards are highly anticipated, especially now that
process that could lead to
these products are appearan international consensus. ing on the market. Yet,
there are no accepted
schemes for comparing
the performance of wearable robots on a vast scale. The only
initiative in this direction is represented by the upcoming Cybathlon competition [4]. In the clinical and biomechanics field,
many metrics and clinical scales have been defined and are regularly used to assess the locomotion functions [5]. Most of
these scales are based on observation by skilled personnel or
defined on a very general level, measuring variables like average
speed or timed up-and-go. With the increasing application of
sensorized and robotic technology in clinics, the expectation
for new quantitative and reliable metrics is rapidly growing.
In this article, as mentioned previously, our goal is to outline a
benchmarking scheme for bipedal locomotion. Our approach
does not aim to compare systems on a global level to see which
one is better but to assess the several aspects of multifacetted performance, allowing a truthful comparison of each feature independently. We envision a scenario in which using this scheme will
encourage collaboration between different research groups toward
the consolidation of standardized benchmarks and experimental
procedures, and we promote its use as a complementary tool to
the competition-based approaches. The scheme presented in this
article is the result of the joint efforts of five European projects, i.e.,
H2R (www.h2rproject.eu), BALANCE (www.balance-fp7.eu),
KoroiBot (www.koroibot.eu), WALK-MAN (www.walk-man.eu),
and BioMot (www.biomotproject.eu). We think that the proposed
scheme can be taken as a starting point for a global iterative process that could lead to an international consensus, based on its
practical use across different laboratories.
Design Approach
Analysis of the Needs: The Web-Based Survey
A benchmark can be considered successful if and only if it is
widely accepted by the community at which it is targeted. To
104

IEEE ROBOTICS & AUTOMATION MAGAZINE

September 2015

reach this goal, a number of key principles for a successful
benchmarking scheme have been identified [6].
● The benchmarks must be well defined, i.e., they really must
serve their purpose. As a consequence, the purpose should
be clear.
● Benchmarks should be rigorously focused on limited, particular subdomains.
● It is more likely that a benchmark is successful within a
scientific (sub)community if it arises from that community itself.
Our design process started with a web-based survey, to
identify the needs of the different users to which the scheme is
addressed. The research communities considered were humanoid robotics, wearable robotics, and human biomechanics.
The latter has been included because of the increasing need to
merge insights from biomechanics and human motor control
in robotic research. The survey (see Figure 1) comprised nine
questions, which, overall, address the first two aforementioned
design principles. The first three questions aimed to collect
general information about the respondents, such as their background and their overall interest in using a benchmarking
scheme and in sharing the data obtained by its use. The last six
questions focused on the contents of the ideal benchmarking
scheme, in terms of
● its general purpose
● the motor functions addressed
● the performance variables to be measured
● the conditions to be included
● its technical properties
● information needed to contextualize the results.
In questions 4-9, the user was asked to give a score from
one to five for each of the predefined options. The results are
represented in Figure 1 in terms of the mean values and standard deviations and divided based on the respondents' backgrounds. A statistical analysis of the similarity across
communities has been performed by one-way analysis of
variance (level of significance p = 0.05). Figure 1 also provides a rough classification of results in three classes, according to mean scores across all communities: items of high
relevance (mean score over four, highlighted in green), items
of medium relevance (mean score over three, shown in orange), and items of low relevance (mean score lower than
three, shown in red). An asterisk indicates the items that presented a significant difference in the responses among the
communities (excluding the "other" group).
Existing Taxonomies for Skills and Abilities
Defining a common nomenclature is a basic purpose for a
successful taxonomy. This is particularly true in our case because the target is multidisciplinary and different terms, like
skill, function, ability, task, activity, action, and performance,
can have different meanings. Inspired by the approach of Magill [7], we will make use of three terms: skill, ability, and performance (see Figure 2). We define skill as a task or activity
with a specified goal. For instance, walking is a motor skill
whose goal is to move from point A to point B. Ability can be

http://www.h2rproject.eu http://www.balance-fp7.eu http://www.koroibot.eu http://www.walk-man.eu http://www.biomotproject.eu

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015