Mistakes in ... Booklet 2020 - 33

ueg education

An example project setting
A fellow has been oﬀered to lead a research project
looking into whether a new bowel preparation
protocol is helpful for polyp detection.

What might happen if there is no clear
research question?
The fellow makes a tremendous eﬀort to obtain the
data describing the polyp findings in patients
undergoing colonoscopy after following the new
preparation protocol, only to then realize that other
essential data have not been collected. For example,
data are missing regarding a control group
(e.g. patients who received the prior standard
protocol), the study's indication (screening versus
diagnosis) and pathological reports (possibly
adenoma detection is a more clinically relevant
outcome).

What could have been done diﬀerently?
A possible research question could have been
"What is the eﬀect of the new bowel preparation
protocol on adenoma detection, compared with
the previous standard protocol, in patients
undergoing screening colonoscopy?"
An alternative question may be framed to see if the
new protocol is "helpful in finding polyps."
Other questions may address patients who
previously had a poor bowel preparation with the
old protocol (diﬀerent patient population),
detection of flat or serrated polyps (diﬀerent
outcomes using endoscopic or histopathological
features), and so on.

Figure 2 | The importance of having a clear research
question.

Clear research questions in epidemiological study
design are often framed using a PICO format, with
a clearly defined patient population, intervention
(or exposure), comparison and outcome.6 Only
once a research question has been framed, can a
clear plan for the study design, data collection and
analysis be completed (further information can be
found in an article we wrote examining this
'mistake' from a Young GI angle6).

Mistake 3 Presuming only a randomized
controlled trial will provide the answer to
your question
The prevailing school of thought in the teaching
and practice of medicine across Europe is that
medicine should be evidence based.7,8 Indeed,
this seems rather intuitive-who would not
want to be treating their patients, or treated
as a patient, on the basis of the best available
knowledge at the time?
Ubiquitous within the teaching and
scholarship of evidence based-medicine (EBM)
is the evidence hierarchy.9 Most people will be
familiar with this as a pyramidal representation
of different study designs, with the 'lowest' form
of evidence provided by case reports and case

Mistakes in... 2020

series at the base, moving up through various
epidemiological study designs, case-control
studies, cohort studies, and randomized
controlled trials (RCTs), with the top of the
pyramid culminating in systematic reviews and
meta-analyses. Following our literature review
and research question refinement, we determine
we need to conduct primary research (thus
removing the option of a systematic review or
meta-analysis) and are left with RCTs as the 'gold
standard' in research.
Trying to push our research into the
highest tier of the EBM pyramid on no other basis
than believing it will therefore provide 'better
quality evidence' is a foolish way to embark
on research.10 Indeed, we recommend that
you examine the question and choose the
appropriate study design. Each epidemiological
study design has its merits and its weaknesses,
and each is better suited to answering
certain types of question.11 In particular, recent
advances in causal inference methodology have
removed some of the previous postulated
weaknesses of observational studies not being
able to answer causal questions.12 Arguably, a
well-conducted causal observational study will
provide better evidence of a real world application than an RCT, conducted as they usually are
on a highly sampled population (see mistake 7).
In addition, there are several pertinent
research questions that can only be answered
by eliciting patient or physician preferences and
viewpoints. Although this article is considering
quantitative epidemiological study designs only,
we cannot stress strongly enough that the findings
of a well-conducted qualitative study are just as
important in adding to our evidence base if the
question being asked is best answered by a study
of this nature. As EBM pioneer David Sackett so
eloquently said more than 20 years ago, "Each
method should flourish because each has features
that overcome the limitations of others when
confronted with questions they cannot reliably
answer."13 Precisely!

Mistake 4 Not working with clearly defined
parameters
Defining key data items such as primary exposure
or intervention, primary outcome and covariates
is clearly important for the data collection and
analysis process.14 If the parameters are not clearly
defined nonuniform and/or unsystematic data
collection results, and the data generated will be
uninformative, or worse, may lead to incorrect
inference (bluntly speaking: garbage in = garbage
out). This point is especially important if multiple
individuals are to be involved in the data
collection process.
While this may be less of an issue if the data
item is an objective measurement (e.g. CRP levels),
even if a time frame is defined (e.g. CRP levels after
3 months of treatment), the situation becomes

more challenging when more objective findings
need to be subjectively evaluated (e.g. severity
of mucosal inflammation demonstrated by
colonoscopy), and especially when the entire
evaluation is based on subjective information
(e.g. wellbeing or quality of life [QoL]). The utility
of subjective measurements should not be
discounted however. There is a clear shift to
including patient reported outcome measures
(PROMs) and patient reported experience
measures (PREMs) alongside 'hard' endpoints to
allow full assessment of healthcare quality.15
Using previously validated scores and
questionnaires (e.g. from the International
Household Survey Network (IHSN) Question Bank
[https://qbank.ihsn.org/]) is a way to ensure the
scientific value of any data collected and goes
some way to meet reviewers' expectations for
publication. Reviewing the relevant literature and
consulting an expert in the field as part of
the protocol preparation process may help to
identify appropriate instruments and scoring
systems. Of note, some may require permission
for use. Using standardized instrumentation is
also beneficial in other ways: a clear methods
section becomes easier to write, and the
possibility that the work can be reproduced and
replicated is increased.16,17

Mistake 5 Failing to account for multiple
comparisons
In the age of electronic records and large
datasets, it is important to keep the focus on
the predefined research question and to be
careful with any additional analysis that may be
performed. One of the pitfalls to avoid is blindly
running statistical tests across all parameters
in a large dataset, hoping to stumble across a
significant finding (again, this is more likely to
occur when time pressured and is sometimes
referred to as 'fishing' or 'p-hacking').17
The reason why running numerous statistical
tests can be a mistake is both logical and
mathematical.18 The basis for many statistical
hypothesis tests is evaluating the chance of
obtaining the observed results compared with
what would have been expected (e.g. based on
the general population or the study's control
group), with a commonly used threshold of 5%
for stating that the observed data are significantly
different to the expected (this is viewed by many
as a mistake in itself,19 but further discussion is
beyond the scope of this article). In other words,
there is a 5% chance of getting results that
appear significantly different from the expected
ones, even though the reality is that there is no
significant difference (this is a type-1 error). The
more statistical tests we run, the more these 5%s
start to add up, and, therefore, when multiple
comparisons are performed the threshold for
determining significance is made more
conservative. The Bonferroni correction is often
21
https://qbank.ihsn.org/
Mistakes in ... Booklet 2020

Table of Contents for the Digital Edition of Mistakes in ... Booklet 2020
