Andrey Malinin

My research focuses on probabilistic modelling, uncertainty estimation and generalisation under distributional shift. My work includes: developing single model uncertainty estimation approaches, such as Prior Networks and Ensemble Distribution Distillation; deriving principled ensemble-based uncertainty estimates for structured prediction tasks, like machine translation, speech recognition and vehicle motion prediction; development of benchmarks for investigating distributional shift. I am keen to progress this research area in the broader ML community. To this end I actively foster cross-disciplinary international collaborations and have organized the `Shifts Challenge' on distributional shift at NeurIPS.

One of the biggest challenges in applying machine learning to real-world tasks is the mismatch between training and deployment data, known as distributional shift. This mismatch can cause a degradation in model performance as the degree of shift increases. Distributional shift is ubiquitous in machine learning and is especially important to be aware of in applications with strict safety requirements.

Ideally, machine learning models should generalize well under arbitrary distributional shift and, if they fail to generalize, these models should indicate that using estimates of uncertainty. The lack of large and diverse datasets with examples of ‘in-the-wild’ distributional shift from a range of industrial tasks makes it difficult to develop such models. Furthermore, until recently, the problem of generalization under distributional shift has received limited attention from the general machine learning community.

Yandex Research, in collaboration with researchers from the Universities of Oxford and Cambridge, organized the [Shifts Challenge](https://research.yandex.com/shifts/) at the NeurIPS 2021 conference to address both issues. The goal was to raise awareness of distributional shift and accelerate the development of robust models capable of providing accurate estimates when navigating unfamiliar situations.

The challenge consisted of two phases — development and evaluation, where participants first created their models and tested them on heldout data, respectively. Fifty participants from 13 countries took part in the development phase. From these 50 participants, 17 teams and individual participants later took part in the evaluation phase of the Shifts Challenge.

First place in the weather prediction was awarded to Ivan Bondarenko from _bond005_, second to Stepan Andreev and Andrey Elnikov from _CabbeanWeather_ and third to Ryoichi Kojima, Guillaume Habault, Roberto Legaspi and Shinya Wada from _KDDI Research_. First place in the motion prediction track was awarded to Alexei Postnikov from _SBTeam_, second to Alexey Pustynnikov and Dmitry Eremeev from _Alexey and Dmitry_, and third to Ching-Yu Tseng, Po-Shao Lin, Yu-Jia Liou from _NTU\_CMLab\_Mira_. An additional honorable mention was awarded to Thomas Gilles, Stefano Sabatin, Dmitry Tsishko, Bogdan Stanciulescu and Fabien Moutarde from _Home_, who produced an interesting solution for motion prediction. The machine translation track did not have winners as no solution managed to beat the benchmark.

At NeurIPS 2021, we ran an online Shifts Challenge breakout workshop and announced the winners, who later [presented their solutions](https://www.youtube.com/watch?v=7Vc6EBNI-kE&list=PLJOzdkh8T5koRCRC6B-H4m1e-iTWrKbYo&index=1) at the poster session. Leading experts in the field gave keynotes and participated in a panel discussion on the following: 

*   [Shifts and other vices of human-computer dialogue](https://www.youtube.com/watch?v=B4c9i8-w1as&list=PLJOzdkh8T5koRCRC6B-H4m1e-iTWrKbYo&index=3) — Nurul Lubis and Michael Heck (Dusseldorf University)
*   [Distributional shift and robustness in Autonomous Vehicle Planning](https://www.youtube.com/watch?v=g8HT4dkojz4&list=PLJOzdkh8T5koRCRC6B-H4m1e-iTWrKbYo&index=4) — Rowan McAllister (Toyota Research)
*   [Adapting to novel data distributions](https://www.youtube.com/watch?v=VpcMNmoABgE&list=PLJOzdkh8T5koRCRC6B-H4m1e-iTWrKbYo&index=5) — Prof. Kate Saenko (Boston University)
*   [The frontiers of research in uncertainty estimation and robustness to distributional shift](https://www.youtube.com/watch?v=TzCaCAhOO_c&list=PLJOzdkh8T5koRCRC6B-H4m1e-iTWrKbYo&index=6&t=3987s), panel discussion — Rowan McAllister, Ramon Astudillo (IBM), Prof. Thomas Dietterich (University of Oregon) and Prof. Sergey Levine (University of Berkeley)

You can find all the videos [here](https://www.youtube.com/playlist?list=PLJOzdkh8T5koRCRC6B-H4m1e-iTWrKbYo).

Distributional shift is the mismatch between training and deployment data that is ubiquitous in the real-world. Studying this phenomenon can enable safer and more reliable ML systems.

Distributional shift

Uncertainty estimation enables detecting when ML models make makes. This is of critical importance in high risk machine learning applications, such as autonomous vehicle and medical ML.

Uncertainty estimation 

One of the biggest challenges in applying machine learning to real-world tasks is the mismatch between training and deployment data, known as distributional shift. This mismatch can cause a degradation in model performance as the degree of shift increases. Distributional shift is ubiquitous in machine learning and is especially important to be aware of in applications with strict safety requirements.

Outcomes of the Shifts Challenge