IEEE Systems, Man and Cybernetics Magazine - April 2022 - 17

Given the dynamic, complex, and cognitively challenging
nature of many mission-critical military systems with
AI-enabled autonomous functions, the IAS framework,
ICD approach, and IMPACTS model have been employed to
support a series of CAF capability and concept development
and evaluation activities. These activities provided
various venues to validate the IAS paradigms for broader
defense and civil applications, such as autonomous transportation,
home care and surgical robots, and Industry 4.0
smart manufacturing [3]-[5]. Meanwhile, the innovative
IAS framework and ICD approach are praised by academic
experts, industrial practitioners, government authorities,
and users from operational communities for their novelty
and trend-setting initiatives of human-AI symbiotic partnership.
The ICD principles are referred to as " a must read
(consideration) for any serious professional in academia,
government, or industry " and an excellent guide to the
design of " 21st century human-computer symbiosis technologies "
[9]. They are noted for setting the agenda for the
coming years as human factors practitioners grapple with
the demands that IASs will make on its operators and outlining
how collaboration and partnership between human
and AI can be achieved through ICD, according to [50]. IAS
broad acceptance, significant impact, and exceptional contribution
have been recognized by the Department of
National Defence, Canada, and the Professional Institute
of the Public Service of Canada with the prestigious Science
and Technology Excellence Award and the President's
Achievement Award in 2020 and 2021, respectively.
Future Work in Symbiotic Human-Robot
Teaming and System Verification Regimes
One of the main strengths of the ICD approach is its systematic
and structured process with stakeholder involvement
and identification of system requirements and
critical decisions with associated tasks for sociotechnical
systems in correspondence domain applications. It
addresses a key challenge of system design: the need to
elucidate, develop, and validate operational requirements
that are obscured by the complexity of human-machine
interactions and of the system itself. This has been demonstrated
with the validation studies on a number of military
systems discussed previously and adopted by NATO standards.
The next step is to systematically integrate the ICD
paradigm into defense policies or acquisition processes
and applying the systems engineering approach for defining,
designing, developing, testing, acquiring, and employing
IAS technologies.
The IMPACTS trust model is a practical, conceptual
component supporting the HMI module in the IAS architecture,
as shown in Figure 1. It is an integral part of the
ICD approach and is being exploited as a systems engineering
analysis and design tool for a trust-management
system (TMS) in a context of soldier-robot teaming
(SRT). Meanwhile, a TMS-related mathematical matrix is
also being developed to measure trust in real time during
human interactions with various autonomous systems.
These mechanisms can then be implemented and integrated
into SRT technologies for a series of military exercises.
Once validated through these operational trials,
these paradigms should make significant impacts on the
systematic design and validation methodology for
enabling trust.
The focus on human-machine trust relationships thus
far has been mostly on the human's trust in the machine.
However, to fully consider the safety property of a human-
machine symbiotic partnership, additional trust relationships
of machine-to-machine and machine-to-human
partnership (i.e., does the machine trust the human's decision
making or judgment?) must be considered. A potential
risk might be a machine's blind trust in human decisions
without knowing the decisions made under the impacts of
logical and emotional trust attributes, such as stress or
bias [42]. From a computation perspective, machine learning
may be rigorous without introducing bias if its algorithm
is trustworthy. That is, cognitive and behavioral bias
is often caused by interference inconsistency between
machines and humans. Hence, overcoming bias potentially
requires bidirectional communication and a comprehensive
mechanism through overlaid interactions.
These mutual trust relationships are therefore suited
for representing a more comprehensive and complete trust
partnership. Thus, additional studies need to be conducted
to understand and develop strategies for managing them
simultaneously in real time. For example, how does a
brain-inspired machine learn to trust or assess confidence
in human judgment or decisions? Or what should be done
when it does not trust the human while the human does
not trust the machine? Trust mediation has yet to be
addressed and should be studied.
Further, legal and ethical issues concerning the use of
highly automated systems have been identified in [2] and
[12], including a sensitive topic when considering safety
and mission-critical weapon systems [6]. These issues
include the possibility that a system with autonomous
functions may purposely and deliberately withhold information
concerning a system failure, malfunction, or error.
The question has been raised as to whether certain
trust repair strategies are ethical, and more work is needed
to address these issues. One area is to integrate both a
TMS for a measure of trust (MoT) and an ethical design
review (EDR) in verification regimes, such as formal systems
engineering processes and industry production standards
for autonomous transportation, home care and
surgical robots, and Industry 4.0 smart manufacturing [15],
[51]. This may include analyses of trust and ethics requirements
during a system design process and then quantitative
measures of the tradeoffs (processing time, memory
use, performance, potential misuse, bias, and so on) during
a test and evaluation process.
Accordingly, a " trust certificate " and/or " ethics certificate "
can be issued if test results are satisfactory through
April 2022 IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE 17
IEEE Systems, Man and Cybernetics Magazine - April 2022

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - April 2022
