What is CrowdSourcing?
CrowdSourcing has been used in many applications to solve a diverse number of problems, from simple to very complex ones. For example, NASA used crowdsourcing to identify craters on Mars through the project called ClickWorkers. They gathered more than 100,000 volunteers who analysed and annotated space photographs of Mars. The annotation task performed by the crowd was virtually indistinguishable from the ones annotated by experts. Click here for more examples of CrowdSourcing tasks.
To use CrowdSourcing as a MetoDHology, you need to design your task in a way that is suitable for the participation of a large number of people (the crowd!). The “secret” of CrowdSourcing is to design tasks in a way that the work produced by the crowd can be easily aggregated.
According to Jeff Howe who coined the term back in 2006, “Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call” [1]. Complementing his definition, in a previous work, Gadiraju and I refer to CrowdSourcing as the outsourcing of well-defined tasks to a multitude of individuals able to perform them [2].
Note that we specify that the crowd must be able to perform the outsourced task. This is crucial, as there may be cases where we need a skilled crowd or a specific group of people (with shared interests, specific age group, etc.). However, in many cases, a diverse crowd can do the job. NASA ClickWorkers is a good example.
Table 1 shows a few tasks that are typically solved using CrowdSourcing [3].
How does CrowdSourcing work?
Samimi et al. [4] explain the CrowdSourcing workflow in a simplified way (see Fig. 1).
Fig 1. The workflow of submitting and completing tasks via crowdsourcing.[4]
The process starts with the requester, responsible for defining and submitting the task to a CrowdSourcing platform (a very popular CrowdSourcing platform is the Amazon Mechanical Turk).
After a task is launched, workers are able to evaluate and select a task from a list of tasks available on the platform. The selection of the task is usually based on (1) the time they will spend to complete the task; (2) the reward they will receive upon completion of the task; and, last but not least, (3) the task requirements.
After the task is completed, the requester will assess (accept/reject) the work and, finally, release the payments to workers via the platform.
The use of a CrowdSourcing platform to mediate this process is very important: (1) crowdSourcing platforms provide templates for common CrowdSourced tasks; (2) help select the most suitable workers to solve the tasks and provide tools to evaluate and ensure high-quality results.
Example using Amazon Mechanical Turk
Click here to access the Amazon MTurk tutorial.
Ethical Considerations
References
[1] J. Howe. The Rise of Crowdsourcing. Wired. Jun’06.
[2] U. Gadiraju; B. P. Nunes. Coleta de dados através de Crowdsourcing em Pesquisas em Informática na Educação. Metodologia de PesquisaCientífica em Informática na Educação: Abordagem Quantitativa. Porto Alegre: SBC, 2020. v. 2, Available at: <https://metodologia.ceie-br.org/livro-2> [Portuguese version].
[3] Gadiraju, U.; Kawase, R.; Dietze, S. A taxonomy of microtasks on the web. In: Proceedings of the ACM conference on Hypertext and social media, 25, 2014, Santiago. New York: ACM. p. 218–223.
[4] Samimi, P.; Ravana, S. D. Creation of reliable relevance judgments in information retrieval systems evaluation experimentation through crowdsourcing: a review. The Scientific World Journal, v. 2014, p. 1 – 13, 2014.
