IEEE Technology and Society Magazine - Winter 2013 - 9

leading edge

Automatic Quality
Management in
Crowdsourcing

T

he recent advancement of web-based collaboration tools and social networking platforms has enabled a new type of pervasive
working environment. People have the ability to create content, perform interactions, and collaborate online by using web 2.0 technologies. Barriers
imposed by geographical expanse are diminishing
due to our increasing technological "connectedness."
Crowdsourcing refers to a web-based collaboration model where tasks are outsourced to an anonymous workforce [4]. It is now possible to utilize the
knowledge of a large number of people participating
in interactions on the web. Task-based crowdsourcing
platforms such as Amazon Mechanical Turk (AMT)1
enable businesses to access the human-power of thousands of people on demand by posting human-task
requests on Amazon's web site.
Similar to cloud computing where computing power is provided on-request as the demand
increases, crowdsourcing platform management
must provide its human workers to crowdsourcing
customers based on their current needs and requirements. However, users have different expectations
when they register to access the platform. Different
from resource management in a cloud computing
environment, with crowdsourcing the capabilities
and skills of the workers are often unknown and may
change based on performed tasks.
Our research effort in crowdsourcing dates back
to 2007 where we introduced Human-Provided Services (HPS) (see [10] and [11]). HPS aims to realize
the vision of large-scale human computation using
web services technology. In this work, we highlight
1

https://www.mturk.com

Digital Object Identifier 10.1109/MTS.2013.2286420
Date of publication: 9 December 2013

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

|

wINTEr 2013

daniel SCHall

some of the key challenges that need to be addressed
before enterprises can access the labor of crowds
in a seamless and automated manner. we propose
a crowdsourcing platform that we call Crowds as a
Service-CaaS. CaaS has a set of built-in capabilities addressing fundamental requirements of online
businesses. Our proposed crowdsourcing platform
follows an auction approach enhanced with skill
management features.

Literature Overview
The notion of crowdsourcing was coined by Howe [4]
and is defined as "the act of taking a job traditionally
performed by a designated agent and outsourcing it
to an undefined, generally large group of people in
the form of an open call." The crowdsourcing paradigm [3] has recently gained increased attention from
both academia and industry, and is even considered
for application in large-scale enterprises [9]. Crowdsourcing offers a convenient way to solve resourceintensive tasks that cannot be processed by software;
typically all kinds of tasks dealing with matching,
ranking, or aggregating data based on fuzzy criteria.
Some concrete examples include relevance evaluation
[1] and ranking of search results [2].
Numerous additional approaches deal with the
seamless integration of crowds into business processes
and information system architectures. CrowdDB [3]
uses human input via crowdsourcing to process queries that neither database systems nor search engines
can adequately answer. Others study algorithms that
incorporate human computation as function calls [7].
One of the largest and most popular crowdsourcing
platforms is Amazon Mechanical Turk (AMT). Some
works including [5] and [6] have already studied the
most important aspects of the AMT user community,
analyzing their structure and properties.
|

9


https://www.mturk.com

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - Winter 2013

IEEE Technology and Society Magazine - Winter 2013 - Cover1
IEEE Technology and Society Magazine - Winter 2013 - Cover2
IEEE Technology and Society Magazine - Winter 2013 - 1
IEEE Technology and Society Magazine - Winter 2013 - 2
IEEE Technology and Society Magazine - Winter 2013 - 3
IEEE Technology and Society Magazine - Winter 2013 - 4
IEEE Technology and Society Magazine - Winter 2013 - 5
IEEE Technology and Society Magazine - Winter 2013 - 6
IEEE Technology and Society Magazine - Winter 2013 - 7
IEEE Technology and Society Magazine - Winter 2013 - 8
IEEE Technology and Society Magazine - Winter 2013 - 9
IEEE Technology and Society Magazine - Winter 2013 - 10
IEEE Technology and Society Magazine - Winter 2013 - 11
IEEE Technology and Society Magazine - Winter 2013 - 12
IEEE Technology and Society Magazine - Winter 2013 - 13
IEEE Technology and Society Magazine - Winter 2013 - 14
IEEE Technology and Society Magazine - Winter 2013 - 15
IEEE Technology and Society Magazine - Winter 2013 - 16
IEEE Technology and Society Magazine - Winter 2013 - 17
IEEE Technology and Society Magazine - Winter 2013 - 18
IEEE Technology and Society Magazine - Winter 2013 - 19
IEEE Technology and Society Magazine - Winter 2013 - 20
IEEE Technology and Society Magazine - Winter 2013 - 21
IEEE Technology and Society Magazine - Winter 2013 - 22
IEEE Technology and Society Magazine - Winter 2013 - 23
IEEE Technology and Society Magazine - Winter 2013 - 24
IEEE Technology and Society Magazine - Winter 2013 - 25
IEEE Technology and Society Magazine - Winter 2013 - 26
IEEE Technology and Society Magazine - Winter 2013 - 27
IEEE Technology and Society Magazine - Winter 2013 - 28
IEEE Technology and Society Magazine - Winter 2013 - 29
IEEE Technology and Society Magazine - Winter 2013 - 30
IEEE Technology and Society Magazine - Winter 2013 - 31
IEEE Technology and Society Magazine - Winter 2013 - 32
IEEE Technology and Society Magazine - Winter 2013 - 33
IEEE Technology and Society Magazine - Winter 2013 - 34
IEEE Technology and Society Magazine - Winter 2013 - 35
IEEE Technology and Society Magazine - Winter 2013 - 36
IEEE Technology and Society Magazine - Winter 2013 - 37
IEEE Technology and Society Magazine - Winter 2013 - 38
IEEE Technology and Society Magazine - Winter 2013 - 39
IEEE Technology and Society Magazine - Winter 2013 - 40
IEEE Technology and Society Magazine - Winter 2013 - 41
IEEE Technology and Society Magazine - Winter 2013 - 42
IEEE Technology and Society Magazine - Winter 2013 - 43
IEEE Technology and Society Magazine - Winter 2013 - 44
IEEE Technology and Society Magazine - Winter 2013 - 45
IEEE Technology and Society Magazine - Winter 2013 - 46
IEEE Technology and Society Magazine - Winter 2013 - 47
IEEE Technology and Society Magazine - Winter 2013 - 48
IEEE Technology and Society Magazine - Winter 2013 - 49
IEEE Technology and Society Magazine - Winter 2013 - 50
IEEE Technology and Society Magazine - Winter 2013 - 51
IEEE Technology and Society Magazine - Winter 2013 - 52
IEEE Technology and Society Magazine - Winter 2013 - 53
IEEE Technology and Society Magazine - Winter 2013 - 54
IEEE Technology and Society Magazine - Winter 2013 - 55
IEEE Technology and Society Magazine - Winter 2013 - 56
IEEE Technology and Society Magazine - Winter 2013 - 57
IEEE Technology and Society Magazine - Winter 2013 - 58
IEEE Technology and Society Magazine - Winter 2013 - 59
IEEE Technology and Society Magazine - Winter 2013 - 60
IEEE Technology and Society Magazine - Winter 2013 - 61
IEEE Technology and Society Magazine - Winter 2013 - 62
IEEE Technology and Society Magazine - Winter 2013 - 63
IEEE Technology and Society Magazine - Winter 2013 - 64
IEEE Technology and Society Magazine - Winter 2013 - Cover3
IEEE Technology and Society Magazine - Winter 2013 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2013
https://www.nxtbookmedia.com