Online Evaluation for Effective Web Service Development

A series of tutorials

Development of the majority of the leading web services and software products today is generally guided by data-driven decisions based on evaluation that ensures a steady stream of updates, both in terms of quality and quantity. Large internet companies use online evaluation on a day-to-day basis and at a large scale. The number of smaller companies using A/B testing in their development cycle is also growing. Web development across the board strongly depends on quality of experimentation platforms. In this tutorial, we will overview state-of-the-art methods underlying everyday evaluation pipelines at some of the leading internet companies.

We invite software engineers, designers, analysts, service or product managers — beginners, advanced specialists, and researchers — to learn how to make web service development data-driven and do it effectively.

Past events

Tutorial at TheWebConf2018

Tutorial at KDD2018

Tutorial at SIGIR2019


— Online evaluation and basics of mathematical statistics
— Foundations of main evaluation methods, such as A/B testing, interleaving, and observational studies
— Case studies: constructing an experimentation pipeline and evaluation metrics; best practices and common pitfalls
— State-of-the-art techniques, including those based on machine learning, for efficient online experimentation
— Open problems and current challenges for research scientists

Detailed program for
Tutorial at TheWebConf2018
Tutorial at KDD2018
Tutorial at SIGIR2019


Roman Budylin

Experimentation Pipeline, Yandex

Alexey Drutsa

Research Department, Yandex

Gleb Gusev

Research Department, Yandex

Eugene Kharitonov

Facebook AI Research

Denis Kulemyakin

Experimentation Pipeline, Yandex

Pavel Serdyukov

Research Department, Yandex

Igor Yashkov

Experimentation Pipeline, Yandex


Slides of the tutorial at TheWebConf2018

Slides of the tutorial at KDD2018

Slides of the tutorial at SIGIR2019

Overview paper of
the tutorial at TheWebConf2018

Overview paper of
the tutorial at KDD2018

Overview paper of
the tutorial at SIGIR2019

Wed Apr 07 2021 16:22:13 GMT+0300 (Moscow Standard Time)