Online Evaluation for Effective Web Service Development

A series of tutorials
Development of the majority of the leading web services and software products today is generally guided by data-driven decisions based on evaluation that ensures a steady stream of updates, both in terms of quality and quantity. Large internet companies use online evaluation on a day-to-day basis and at a large scale. The number of smaller companies using A/B testing in their development cycle is also growing. Web development across the board strongly depends on quality of experimentation platforms. In this tutorial, we will overview state-of-the-art methods underlying everyday evaluation pipelines at some of the leading internet companies.

We invite software engineers, designers, analysts, service or product managers — beginners, advanced specialists, and researchers — to learn how to make web service development data-driven and do it effectively.

Past events

Tutorial at TheWebConf2018
 
Tutorial at KDD2018


Tutorial at SIGIR2019


Program

— Online evaluation and basics of mathematical statistics
— Foundations of main evaluation methods, such as A/B testing, interleaving, and observational studies
— Case studies: constructing an experimentation pipeline and evaluation metrics; best practices and common pitfalls
— State-of-the-art techniques, including those based on machine learning, for efficient online experimentation
— Open problems and current challenges for research scientists

Detailed program for
Tutorial at TheWebConf2018
Tutorial at KDD2018
Tutorial at SIGIR2019

Speakers

Roman Budylin
Experimentation Pipeline, Yandex
Alexey Drutsa
Research Department, Yandex
Gleb Gusev
Research Department, Yandex
Eugene Kharitonov
Facebook AI Research
Denis Kulemyakin
Experimentation Pipeline, Yandex
Pavel Serdyukov
Research Department, Yandex
Igor Yashkov
Experimentation Pipeline, Yandex

Materials

Slides of the tutorial at TheWebConf2018
 
Slides of the tutorial at KDD2018
 
Slides of the tutorial at SIGIR2019
 
Overview paper of
the tutorial at TheWebConf2018
Overview paper of
the tutorial at KDD2018
 
Overview paper of
the tutorial at SIGIR2019
 
Wed Apr 07 2021 16:22:13 GMT+0300 (Moscow Standard Time)