IEEE Technology and Society Magazine - June 2021 - 60

Feature
understand against which metrics the learning
algorithm is optimized is of paramount importance.
Understanding, for instance, whether a diagnostic
system is optimized to maximize accuracy or minimize
false-positives or a function of both is important
for patients to know and, in some instances,
should be regulated. In some cases, this checkpoint
might let us glimpse problems of transparency in
the algorithmic objectives. For example, in a recommendation
system for an online travel agency,
the success criteria can include the expected sales
commission and not only the fit to user travel preferences.
In that case, this optimization criterion
should at least be known to customers.
* The decision process should be understood and
classified at least in terms of its level of automation.
Even if there is a human decision in the
loop, the actual process of human decision can
be analyzed to ascertain whether the human
decision is independent or can be biased toward
the machine suggestions. In some cases, like in
some justice administration, it could be more
prudent to treat machine suggestions as a way
to contrast judge decisions, to avoid human bias
toward machine-generated recommendations.
* The decision itself should be analyzed in terms of
repercussions on human rights of stakeholders. The
analysis should cover the degree in which these
decisions could affect properties, health, or rights
of individuals or communities. These individuals
or communities could be the users of the system,
but could be indirectly affected nonusers. Here, it
would be relevant to assess the risk of the decisions
in terms of number and groups of people affected,
the nature of potential hazards (reversible/irreversible),
or economic value, to name a few.
* Decisions affect people and since citizens are free
agents, they can change their behavior accordingly.
Individual behavior changes can have community
consequences. Here, it is important to distinguish
different potential opportunities and risk and trade
them off. For instance, there are obvious individual
benefits on personalized content generation but a
continuous exposition to it could build a " bubble
filter " that could harm the diversity of information
and degrade the ability to make well-informed decisions,
for instance, in political choices, additionally
deriving in potential polarization of public opinion.
In this very point, transparency should also be analyzed:
do users, or affected individuals, know that
60
these type of content decisions are being made
derived from user navigation data?
* The data obtained from individuals must be
respectful of Regulation (EU) 2016/679 (General
Data Protection Regulation-GDPR) for a start,
but due to the closed-loop nature of these systems,
data collected from individuals is dynamic and
special requirements on data obsolescence shall
be analyzed. In this sense, the GDPR is requesting
consent from the data subject for automatic decision-making,
but is not requesting from processors
to disclose the description of the decision process.
This specific aspect of GDPR should be reviewed.
As discussed earlier, it is very important to include
into the system boundary the environment that the AI
system interacts with. Because as we see in this simple
schema, there are feedback loops whose dynamics
need to be taken into account: If we were just to take
into account the specific ML system, you would just
take the training set and learning algorithm and test the
model against accuracy or fairness measures, check for
biases in the training set, even the appropriateness of
success criteria, and so on. But we think it is imperative
that the complete system should also include the decision
process, these decision implications in individual
and, by aggregation, in groups or communities and how
these societal changes may affect back the performance
of the algorithm creating a feedback loop. We believe
that systems thinking approach in a socio-technical environment
is an interesting approach and could help in
framing the analysis within the future AI regulation.
Further technical definitions should be made available
since the nature of AI is mathematical and there
might be some concepts that are lost in translation
when trying to formalize them. For instance, the concept
of safety in AI/ML systems have to be redefined
with respect to other products. Most of the current
AI applications are developed with the objective of
classify or predict, based on algorithms of statistical
inference. As such, they are subject to statistical errors,
such as false-positives or -negatives in a binary classification
system. In this respect, the safety of the system
should be defined as a statistic parameter range.
Explainability, in many cases, is not as necessary, as
long as there is a way to audit results with synthetic or
real data and prove that these parameters are within
safety limits provided by the specific application regulation.
Those limits should be known and certified, so
users and stakeholders, in general, are aware of these.
IEEE Technology and Society Magazine
IEEE Technology and Society Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - June 2021
