IEEE Technology and Society Magazine - Winter 2013 - 12

Rate

- Task Processing
- Crowd Capacity
- (Mis-)Behavior

Monitoring

Task

Invitation
Bid

Auctioning
- Matching
- Ranking
- Selection

Task

User Management
- Profiles
- Ratings
- Observations

Policies
Producer Interface

7 Receive

- Quality
- Skill Evolution
- Profit

Result

Consumer

Result

Rating

Offer

Consumer Interface

Crowd Platform Management
Invite

Bid

Producer

4 Assign

6 Complete

5 Process

Fig. 1. architecture of the CaaS crowd-sourcing platform.

task to the top-ranked worker. After the processing
of the task the producer submits the result to CaaS.
CaaS then forwards it to the consumer. Finally, the
consumer is given the opportunity to issue a rating of
the transaction. The rating data is used to update the
producer's track record.
Internally, CaaS is based on four core components.
At the heart of the platform the Auctioning component is responsible for bringing together producers and
tasks. A matching algorithm calculates the suitability
of producers for a given task. This is used to invite a
number of users to an auction. A rating mechanism
allows for the ranking of bids according to suitability and monetary aspects. The selection functionality
selects zero to multiple (but usually one) producers
and assigns the task to them. For each producer the
User Management contains a profile consisting of a
user-provided description of skills and interests, the
ratings a user has received, and finally historical
information describing the producer behavior, such as
timeliness of task processing.
The Monitoring component gathers information
relevant to the system, in particular, data about task
processing (un-assignable tasks, delayed tasks, etc.),
statistics about the availability of skills and workload
of the crowd, and the behavior of members, which,
among others advantages, allows identification of
malicious members. Policies are used for goal-driven
management of CaaS specifying the expected quality of task-processing and profit considerations. Also,
when certain skills become sparse, policies are used
to control counteractions.

Simulation of Crowds and Task Markets
we have implemented a Java-based simulation framework to evaluate different features of our CaaS approach.
At this stage, the goal of our implemented framework
was to support the full lifecycle of auction-based task
processing. The framework supports the simulation of
12

consumer and producer behavior. The simulations are
performed by creating consumers that offer a number
of tasks to be processed by a pool of producers (workers). The consumers' offering behavior depends on the
quality of returned task results. Consumers stop offering tasks if producers deliver low quality results. As a
consequence, the platform's profit decreases and producers may remain idle because of the limited number
of tasks available on the market.
we have tested the skill evolution feature of the platform to ensure high quality task results. The platform
automatically performs skill evolution through training
tasks that are assigned to newcomers and inexperienced
users. These training tasks are automatically created by
the platform in the context of an auction. The winner of
an auction performs the task whose result is returned to
the consumer. The training task is performed by a newcomer but the task result is only used by the platform.
The platform compares both results to estimate the skill
of the newcomer based on the result of the experienced
user and the inexperienced user. Training tasks are only
created if the workload of users is low to avoid delays
and assignment problems of real tasks. This procedure
has two advantages. The platform is able to maintain an
up-to-date-view on the skills of people. Also, newcomers and inexperienced users receive training that is paid
by the platform. This has the advantage that consumers
do not need to manually create training or qualification
tasks as this is already performed by CaaS.
Our experiments in which we have focused on skill
evolution show very good results such as increasing
consumer satisfaction and ratings, increasing platform
profit, and also worker profit. A web-based demo is
available online (http://tinyurl.com/CrowdSim).

Future Work
In our future work we will analyze the evolution of communities (birth, expansion, contraction, and death) by
looking at the task posting behavior of requesters. This
IEEE TECHNOLOGY AND SOCIETY MAGAZINE

wINTEr 2013

http://www.tinyurl.com/CrowdSim

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - Winter 2013