We look forward to sharing our research and best practices at The Web Conference 2021. In our hands-on tutorial, we present a systematic view on using Human-in-the-Loop to obtain scalable offline evaluation processes and, in particular, high-quality relevance judgements.
We will introduce the ranking problem to the attendees, discuss the commonly used ranking quality metrics, and then focus on Human-in-the-Loop-based approach to obtain relevance judgements at scale. More precisely, we will present a thorough introduction to pairwise comparisons, demonstrate how these comparisons can be obtained using Crowdsourcing, and organize a hands-on practice session in which the attendees will obtain high-quality relevance judgements for search quality evaluation. Finally, we will discuss the obtained relevance judgements, point out directions for further studies, and answer questions asked during the tutorial. We look forward to seeing you at our tutorial!
“Improving Web Ranking with Human-in-the-Loop: Methodology, Scalability, Evaluation” by Alexey Drutsa, Dmitry Ustalov, Nikita Popov, Daria Baidakova
Tuesday, April 13, 2021, 9:00am-3:00pm (Online)