Crowd Science Workshop:

Remoteness, Fairness, and Mechanisms
as Challenges of Data Supply by Humans for Automation

Workshop at
NeurIPS 2020


Fri, Dec 11th  
or Sat, Dec 12th

Despite the obvious advantages, automation driven by machine learning and artificial intelligence carries pitfalls for the lives of millions of people. The pitfalls include disappearance of many well-established mass professions and increasing consumption of labeled data produced by humans. Those data suppliers are often managed by old fashioned approach and have to work full-time on routine pre-assigned task types. Crowdsourcing methodology can be considered as a modern and effective way to overcome these issues since it provides flexibility and freedom for task executors in terms of place, time and the task type they want to work on. However, many potential stakeholders of crowdsourcing processes hesitate to use this technology due to a series of doubts (that have not been removed during the past decade). In order to overcome this, we organize this workshop which will focus research and industry communities on three important aspects: Remoteness, Fairness, and Mechanisms.

Remoteness. Data labeling requesters (data consumers for ML systems) doubt the effectiveness and efficiency of remote work. They need trustworthy quality control techniques and ways to guarantee reliable results on time. Crowdsourcing is one of the viable solutions for effective remote work. However, in spite of the rapid growth and the body of literature on the topic, crowdsourcing is in its infancy and, to a large extent, is still an art. It lacks clear guidelines and accepted practices for both the requester and the performers (also known as workers) side, which significantly impedes the opportunity to realize the full potential of crowdsourcing. We intend to end this trend and achieve a breakthrough in this direction.

Fairness. Crowd workers (data suppliers) doubt the availability and choice of tasks. They need fair and ethical task assignment, fair compensation, and growth opportunities. We believe that a working environment (e.g., a crowdsourcing platform) may help here since it should provide flexibility in choosing/switching tasks and working hours, as well as act fairly and ethically in task assignment. We also aim to address bias in the task design and execution that can skew results in ways that had not been anticipated by data requesters.

Since quality, fairness and growth opportunities for performers are central to our workshop, we will invite a diverse group of performers from a global public crowdsourcing platform to our panel-led discussion.

Mechanisms. Matchmakers (the side of the working environment, usually represented by a crowdsourcing platform) doubt the effectiveness of economic mechanisms that underlie their two-sided market. They need such mechanism design that guarantees proper incentives for both sides to provide flexibility and fairness for workers, while quality and efficiency for data requesters. We stress that the economic mechanisms are the key to successfully address the issues of remoteness and fairness. Hence, we intend to deepen the interaction of communities that work on mechanisms and crowdsourcing.

Invited speakers
Lora Aroyo
Google Research NYC, USA
Gianluca Demartini
University of Queensland, Australia
Panos Ipeirotis
New York University, USA
Matt Lease
University of Texas at Austin, USA
Olga Megorskaya
Toloka, Russia
Seid Muhie Yimam
Universität Hamburg, Germany

Program Committee

Marcos Baez, University of Trento
Boualem Benatallah, The University of New South Wales
Alessandro Bozzon, Delft University of Technology
Alessandro Checco, The University of Sheffield
Anna Lisa Gentile, IBM
Gleb Gusev, Sberbank Evgeny Krivosheev, University of Trento
Alexey Kushnir, Carnegie Mellon University Tepper Business School
Lucas Maystre, Spotify
Svetlana Nikitina, University of Trento
Ivan Stelmakh, Carnegie Mellon University
Jie Yang Delft, University of Technology
Fedor Zhdanov, AWS
Xiong Zhou, University of Houston

 Key dates

All deadlines are at 23:59 AOE  

Paper submission deadline: Oct 09, 2020 
Accept/Reject notification: Oct 30, 2020 
Camera-ready papers due: Nov 15, 2020
NeurIPS conference: Dec 6 - Dec 12, 2020

Registration and participation

For registration and pricing see Registration for the conference is mandatory. 

If you would like to participate in our workshop RSVP using the form below and we will send you a reminder about the registration for NeurIPS 2020 and will share the details closer to the day of the workshop.

Daria Baidakova
Fabio Casati
Alexey Drutsa
Dmitry Ustalov


If you are interested in helping out with the review process or you have other questions, please get in touch with us:
Mon Oct 26 2020 11:27:26 GMT+0300 (Moscow Standard Time)