Talking Stick - January/February 2016 - 29

"Surveys are a wonderful tool to
gain understanding; however, poorly
structured questionnaires can derail
the whole purpose," says Lambrect.
When evaluating learning, for
example, it is tempting to ask students
simple questions such as "How much,
on a scale of 1 to 5, did you learn as
a result of this program?" However,
this question might not provide
the best results, as students tend to
misrepresent their own learning. Also,
without any other questions to use
as comparison points, what do the
categories really mean? Did a student
who checks a 4 actually learn more
than their peer who checked a 3?
Use a multidimensional approach
when formulating survey questions.
"If there is something you are really
interested in understanding, you may
want to ask a number of questions
that consider multiple points of view
in order to get a complete depiction,"
suggests Daphna Harel, assistant
professor of applied statistics at
NYU. "Single questions can be
easily misconstrued. You want to
avoid having important questions
taken in different ways by different
individuals." An example of this
approach can be found in the SILLP
survey, launched toward the end of
2015 through NYU Associate Professor
Matthew J. Mayhew's College
Impact Laboratory (CoIL). Students
were assessed using questions that
addressed the influence of residential
experiences on attributes related to
learning, such as academic confidence,
academic progress, critical thinking,
and utilization of academic supports.
As described here, several questions
on the survey instrument addressed
each dimension in order to meet the
stated outcomes.
Next, ask, "Who can best answer
my question?" Depending on what you
are attempting to learn, be intentional
about who receives the survey. Make
sure your "what" is clearly stated in

your outcome, e.g., "As a result of this
survey, I want to know how students
in science, technology, engineering,
and mathematics (STEM) perceive the
academic support provided in their
field-specific living-learning program."
Keeping your target population clear
will help keep your survey questions
relevant.
The kind of demographic
information that needs to be
collected depends on the question
being asked. Avoid requesting
demographic information you don't
actually need. "Remember to only ask
those demographic questions most
important to you. Sometimes, to get
the number of responses you need,
shorter is better as it is less tedious
for students," Mayhew advises. For
the STEM example, a simple question
on the students' living arrangement
- traditional residence hall or livinglearning program - might cover all the
demographic information needed. If
you are interested in understanding
students' social identities (categorized
by gender, by sexual orientation, by
religion, etc.), then you should ask
these questions using the most up-todate options; if you are not, don't.
Finally, you need at least 20
participants from each group to
calculate meaningful statistics. "If your
groups are smaller, adding open-ended
prompts, or even facilitating focus
groups, may help you deepen your
understanding of the how and why of
student responses since you shouldn't
statistically compare them," suggests
Mayhew.
Who else will care about this
information, and what will they want
to know? Identifying the survey's
stakeholders can sometimes be
overwhelming: So many people want
a piece of the pie! For example, the
stakeholders of our STEM livinglearning program may include the
residence life director (who is most
interested in student interactions

with the RAs) and the partnering
academic dean (who is most interested
in students' academic outcomes).
Without corralling all of these voices,
you could end up with a really long
survey that students will not finish or
too many surveys that the students will
not even open.
However tempting it may be, Harel
advises against writing a survey in
isolation. You need the input of all the
stakeholders to make decisions about
the "what" and "who" portions of the
survey. By bringing the stakeholders
together, you and they are able to see
the variety of motivations for surveying
and decide which information is most
important. This is also where you can
combat survey fatigue by streamlining
the instruments and ensuring that
you're capturing the appropriate
respondents for each survey.
One way that SILLP has found
that middle ground is by determining
the most common stakeholders for
living-learning programs and asking
questions in a way that creates relevant
information for everyone. If our
STEM living-learning program were
surveyed, the respondents would
answer questions about student
interactions in the residence hall
(for the residential life director) that
influence academic development (for
the partnering academic dean). By
using this survey design technique,
students are asked a reasonable
number of questions and the various
stakeholders receive information
useful to them.
Next, consider what resources for
post-survey analysis you have. Maybe
there is a bona fide statistician on
your staff. Or maybe there's someone
on your team who likes working
with numbers. But while these two
scenarios are both useful for an
assessment plan, they might suggest
different final products. Knowing what
you can do with your data set is part
of knowing what kind of data set you

JANUARY + FEBRUARY 2016

29
Talking Stick - January/February 2016

Table of Contents for the Digital Edition of Talking Stick - January/February 2016
