Improving Web Ranking 
with Human-in-the-Loop: 
Methodology, Scalability, Evaluation

Tutorial at TheWebConf 2021 (WWW '21)

Room: Online
Starts: 09:00

Date: April 13, 2021

Modern Web services widely employ sophisticated Machine Learning techniques to rank news, posts, products, and other items presented to the users or contributed by them. These techniques are usually built on offline data pipelines and use a numerical approximation of the relevance of the demonstrated content. In our hands-on tutorial, we present a systematic view on using Human-in-the-Loop to obtain scalable offline evaluation processes and, in particular,  high-quality relevance judgements. We will introduce the ranking problem to the attendees, discuss the commonly used ranking quality metrics, and then focus on Human-in-the-Loop-based approach to obtain relevance judgements at scale. More precisely, we will present a thorough introduction to pairwise comparisons, demonstrate how these comparisons can be obtained using Crowdsourcing, and organize a hands-on practice session in which the attendees will obtain high-quality relevance judgements for search quality evaluation. Finally, we will discuss the obtained relevance judgements, point out directions for further studies, and answer questions asked during the tutorial.


Alexey Drutsa

Crowdsourcing Department, Yandex

Dmitry Ustalov

Crowdsourcing Department, Yandex

Nikita Popov

Search Department, Yandex

Daria Baidakova

Crowdsourcing Department, Yandex

Conference time zone is UTC+2 (CEST).

Introduction to Offline Evaluation (09:00–09:30)

Ranking and Quality Metrics (09:30–10:30)

Human-in-the-Loop Essentials (10:30–11:15)

Pairwise Comparisons (11:15–12:15)

Hands-On Practice Session I (12:15–13:00)

Lunch Break (13:00–14:00)

Hands-On Practice Session II (14:00–15:45)

Final Remarks and Conclusion (15:45–16:00)

Wed Jun 02 2021 17:49:33 GMT+0300 (Moscow Standard Time)