Crowd Science Workshop: 

Remoteness, Fairness, and Mechanisms
as Challenges of Data Supply by Humans for Automation

Workshop at
NeurIPS 2020

08:00 - 16:00 PST

Fri, Dec 11th

Call for Papers

We welcome papers on the broad range of topics (see below). There are two types of submissions:

— «regular» papers that provide a validated scientific contribution to crowd science;
— work-in-progress or vision papers that propose ideas or present preliminary results.

In both cases, a paper can provide research contributions (theoretical and applied) or can describe designs and implementations of solutions/systems for practical tasks in some industry. In the latter case, papers solve or advance the understanding of ​issues related to deploying technologies based on human data supply for automation in the real world.

In all cases, a submitted paper must describe new, unpublished research, and must not have been published or under review elsewhere. 


We invite high-quality submissions one or more of the following areas in crowd science (both theoretical and practical contributions are welcome). Our (non-exhaustive) list of topics of interest include:

1. Effectiveness and efficiency of remote work:  
— Trustworthy techniques for quality control and assurance, including but not limited to the ones used in crowdsourcing
— Extraction of reliable data, including but not limited to: aggregation methods, post-verification, etc
— Decomposition of tasks and pipelines of them (complex workflows)
— Design of tasks and instructions
— Reputation and trust systems
— Human psychology of learning and behavioral sciences

2. Fairness: 
— Fair and ethical task assignment
— Fair pricing and overall compensation for crowd workers
— Growth opportunities for crowd workers
— Recommendation systems for task assignment

3. Mechanisms:
 — Two-sided markets, market design, labor markets, data markets, prediction markets, etc
— Pricing and incentives, including but not limited to: surge pricing, dynamic pricing, etc
— Auction design for crowdsourcing, labor markets, and dataset markets
— Matching and task allocation, including weak matching (e.g., via recommendations and freedom of choice)
— Matching for spatial tasks
— Cooperative games
— Mechanisms for privacy, fairness, and ethics

4. Automation with data supply powered by humans: 
— Human-in-the-loop techniques, including but not limited to: active learning, interactive learning, online learning, etc
— Real-time crowdsourcing
— Collection, preparation, and management of data for ML and AI
— AI systems with explainability and interpretability

5. Topics of special interest: 
— Crowdsourcing: from art to engineer science (including but not limited to replicability of crowdsourcing and human computation experiments)
— Methodological development of crowdsourcing
— Psychological safety of performers working with violent and/or explicit content
— Local specificity and the importance of context for task execution
— Bias in the task design and execution

6. Other:
— Perspectives on the future of crowdsourcing
— Applications of crowdsourcing techniques, including but not limited to: QA, content moderation, spatial crowdsourcing, etc
— Crowdsourcing platform design and development
— Automation for data supply, including but not limited to: unsupervised learning for data labeling, weak learning, etc
— Crowdsourcing practice  

Program Committee

Marcos Baez, University of Trento
Boualem Benatallah, University of New South Wales
Alessandro Bozzon, Delft University of Technology
Alessandro Checco, University of Sheffield
Anna Lisa Gentile, IBM
Gleb Gusev, Sberbank
Evgeny Krivosheev, University of Trento
Alexey Kushnir, Carnegie Mellon University
Anna Lioznova, Yandex
Lucas Maystre, Spotify
Svetlana Nikitina, University of Trento
Maria Sagaidak, Yandex
Ivan Stelmakh, Carnegie Mellon University
Jie Yang, Delft University of Technology
Fedor Zhdanov, Amazon
Xiong Zhou, Amazon

Key dates

All deadlines are at 23:59 AOE  

Paper submission deadline: Oct 09, 2020 
Accept/Reject notification: Oct 30, 2020
Talk recordings: Nov 14, 2020
Final papers due: Nov 30, 2020
NeurIPS conference: Dec 6–12, 2020

Submission instructions and format

Papers should be submitted via EasyChair by the end of the deadline day Oct 02, 2020 (AOE).
Paper submission portal 

All submissions must be in PDF format. The page limit is up to eight (8) pages maximum for regular papers and four (4) pages for work-in-progress/vision papers. These limits are for main content pages, including all figures and tables. Additional pages containing appendices, acknowledgements, funding disclosures, and references are allowed. You must format your submission using the NeurIPS 2020 LaTeX style file which includes a «preprint» option for non-anonymous preprints posted online. The maximum file size for submissions is 50MB. Submissions that violate the NeurIPS style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review. 

As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information. 

Review process

Each paper will be reviewed by at least three members of the program committee. Reviewers will be instructed to evaluate paper submissions according to specific review criteria. We encourage authors to review them before submission.

Review criteria are:
— Relevance
— Readability/clarity
— Originality
— Technical correctness/soundness
— Reproducibility
— Substance
— Potential impact on the industry 

To ensure relevance, authors should consider including research questions and contributions of broad interest to the topic of the workshop, as well as discuss relevant open problems and prior work in the field. 

The reviewing process will be double blind at the level of reviewers (i.e., reviewers cannot see author identities) but not at the level of workshop organizers. As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information (even in the supplementary material).

To ensure fairness, we prevent the PC members from reviewing submissions declared as conflicts-of-interest with them. 

Publication and proceedings

First, accepted papers will be listed on the workshop website. Second, proceedings shall be submitted to for online publication (alternatively, the authors can publish the preprints on arXiv).

To be included in the proceedings and in the workshop program at least one author must register for the main conference (NeurIPS 2020) and present the accepted work at the workshop.

Presenting your paper

As noted earlier, at least one author of each accepted paper must register for the main conference (NeurIPS 2020) to present the work or acceptance will be withdrawn.

Note: NeurIPS 2020 is a virtual-only conference.


If you are interested in helping out with the review process or you have other questions, please get in touch with us:

Wed Apr 07 2021 16:19:04 GMT+0300 (Moscow Standard Time)