Instrumentation & Measurement Magazine 24-6 - 30

Study Design
In this paper, we report on the results of a retrospective correlational
study, henceforth referred to as " the Vibrant Minds
study. " During this study, participants played the three serious
games described above over eight weeks. Throughout this
time, we collected a number of repeated measures, described
in detail below. We fitted a multilevel linear mixed model
(MLMM) to explain the evidence from longitudinal data on
the engagement experienced by the older adults participating
in our study, while playing the games.
Data Source-Participants
The study participants were 14 older adults, some who were
cognitively healthy (n= 9; 64.3%) and some with mild or moderate
dementia (n= 5; 35.7%). The mean Montreal Cognitive
Assessment (MoCA) [10] for the entire population was 24.43
(SD = 6.03) (mean MOCA Older adults with dementia = 18.25
(SD=7.00); mean MOCA Older adults without dementia=25.80;
SD=4.10; [F(1, 13) = 6.51, p = 0.025]. The participants were primarily
female (n = 10; 71.4%), the mean age was 76.48 years
(SD = 8.63 years), and most lived in the community with others
(n=11; 78.5%) and had a high school diploma (n=6; 42.8%).
Intervention and Procedures
This study was approved by the University of Alberta, Edmonton,
AB, Canada ethics board (Pro00069138). Written consent
was obtained from participants or their substitute decision
makers. Participants played a game for 30 minutes during 16
sessions over a period of eight weeks. Each participant was
randomly assigned to one of the games and played the same
game on a tablet for all 16 sessions. Every session was conducted
in a room where the participants were seated at a table.
In addition, participants had the opportunity to watch a short
demonstration video showing step-by-step instructions on
how to play the mobile games. The Positive and Negative Affect
Schedule (PANAS) [11] and engagement questionnaires
were administered at the end of each session.
Data, Variables, and Measures
The study data set consists of 224 data points collected during
the 16 gameplay sessions. The variables were grouped
into domains, namely, participants' demographics (e.g.,
gender, cognitive status), participants' game performance
(e.g., game level achieved), type of games played, participants'
affect while playing the games, technical problems,
and environmental disturbances during gameplay. Table 1
describes the data in more detail. The Engagement score was
calculated by adding up the score of each item, resulting in
a total value between 8 and 32. The higher the engagement
score, the more engaged the older adult was while playing
the mobile game.
The Affect variable was calculated as the difference of the
sum of the values of the negative affect (e.g., distressed, upset,
and ashamed) subscale from the sum of the values of the
positive affect (e.g., interested, excited, and active) subscale of
the PANAS score, resulting in a number between -50 and 50
30
(positive values indicate a positive affect; negative values indicate
a negative affect).
Data Analysis
Principal Component Analysis (PCA) with varimax rotation
was used to reduce the dimensionality of the dataset. The eigenvalue,
a measure of how much variance of the observed
variables a factor explains, was set to a cut-off value of 1.0.
We used a MLMM for the longitudinal measures to determine
the effects of time (sessions) and other time-varying (e.g.,
participants' performance, affect) and time-invariant (e.g.,
gender) covariates on the engagement score. The MLMM is
suitable in longitudinal studies where the subjects are measured
repeatedly over time [12], and it attributes the variance
of the collected measures across a number of different levels
(in our case, there were two levels). This model allowed us to
investigate not only the within-subject variance (Level1, or the
time level), but also the amount of between-subject variance
(Level2, or the subject level, our unit of analysis) in the effects
of the covariates on the subjects in a longitudinal study.
We calculated Intraclass Correlation Coefficients (ICC) to
examine the similarity (or homogeneity) of the responses on
the engagement score within our unit of analysis. In (1), the
ICC quantifies the percentage of variance in the engagement
score that goes to the within-subject (σ2
ject (σ2
within) and between-subbetween
)
components, regardless of time:

2
ICC 
22
between

between

within
If the ICC value was less than 5% in the engagement score
between groups, there was no reason to implement a MLMM,
as the subjects were not different from each other [12]. We used
the Akaike information criterion (AIC), Bayes information criterion
(BIC), and -2 RE/ML log-likelihood (also referred to as
fit criteria) statistics to assess the fit of a model based on its optimum
log-likelihood value. We used the " smaller is better "
form for the information criteria as an indicator of a " better " fit
of the MLMM [12].
In this study, we ran five MLMMs, namely, MLMM A, B,
C, D, and E. Model A (the Null model) was designed to determine
whether there was any significant variance or variability
in the engagement scores between the older adults, regardless
of time. Model B (adding time) aimed to explore whether
the engagement scores changed over the number of sessions
(time). Model C (adding the time-varying covariates at Level
1) aimed to determine whether the covariables (fixed effects,
e.g., game level, PANAS) measured throughout the 16 sessions
had any effect on the engagement scores. Model D (adding
the time-invariant covariates at Level 2 to Model C) aimed
to determine whether the variables (fixed effects, e.g., gender,
cognitive impairment) measured throughout the 16 sessions determined
the engagement scores. Model E was formulated to
determine whether the objective independent measures (e.g.,
game level achieved, and disturbances experienced during gameplay)
contributed to explaining the engagement scores reported
by the participants. In doing so, we withdrew the objective
IEEE Instrumentation & Measurement Magazine
September 2021
(1)
Instrumentation & Measurement Magazine 24-6

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-6
