Outcomes of the Shifts Challenge
One of the biggest challenges in applying machine learning to real-world tasks is the mismatch between training and deployment data, known as distributional shift. This mismatch can cause a degradation in model performance as the degree of shift increases. Distributional shift is ubiquitous in machine learning and is especially important to be aware of in applications with strict safety requirements.
Ideally, machine learning models should generalize well under arbitrary distributional shift and, if they fail to generalize, these models should indicate that using estimates of uncertainty. The lack of large and diverse datasets with examples of ‘in-the-wild’ distributional shift from a range of industrial tasks makes it difficult to develop such models. Furthermore, until recently, the problem of generalization under distributional shift has received limited attention from the general machine learning community.
Yandex Research, in collaboration with researchers from the Universities of Oxford and Cambridge, organized the Shifts Challenge at the NeurIPS 2021 conference to address both issues. The goal was to raise awareness of distributional shift and accelerate the development of robust models capable of providing accurate estimates when navigating unfamiliar situations.
The challenge consisted of two phases — development and evaluation, where participants first created their models and tested them on heldout data, respectively. Fifty participants from 13 countries took part in the development phase. From these 50 participants, 17 teams and individual participants later took part in the evaluation phase of the Shifts Challenge.
First place in the weather prediction was awarded to Ivan Bondarenko from bond005, second to Stepan Andreev and Andrey Elnikov from CabbeanWeather and third to Ryoichi Kojima, Guillaume Habault, Roberto Legaspi and Shinya Wada from KDDI Research. First place in the motion prediction track was awarded to Alexei Postnikov from SBTeam, second to Alexey Pustynnikov and Dmitry Eremeev from Alexey and Dmitry, and third to Ching-Yu Tseng, Po-Shao Lin, Yu-Jia Liou from NTU_CMLab_Mira. An additional honorable mention was awarded to Thomas Gilles, Stefano Sabatin, Dmitry Tsishko, Bogdan Stanciulescu and Fabien Moutarde from Home, who produced an interesting solution for motion prediction. The machine translation track did not have winners as no solution managed to beat the benchmark.
At NeurIPS 2021, we ran an online Shifts Challenge breakout workshop and announced the winners, who later presented their solutions at the poster session. Leading experts in the field gave keynotes and participated in a panel discussion on the following:
- Shifts and other vices of human-computer dialogue — Nurul Lubis and Michael Heck (Dusseldorf University)
- Distributional shift and robustness in Autonomous Vehicle Planning — Rowan McAllister (Toyota Research)
- Adapting to novel data distributions — Prof. Kate Saenko (Boston University)
- The frontiers of research in uncertainty estimation and robustness to distributional shift, panel discussion — Rowan McAllister, Ramon Astudillo (IBM), Prof. Thomas Dietterich (University of Oregon) and Prof. Sergey Levine (University of Berkeley)