AM

Andrey Malinin

My research focuses on probabilistic modelling, uncertainty estimation and generalisation under distributional shift. My work includes: developing single model uncertainty estimation approaches, such as Prior Networks and Ensemble Distribution Distillation; deriving principled ensemble-based uncertainty estimates for structured prediction tasks, like machine translation, speech recognition and vehicle motion prediction; development of benchmarks for investigating distributional shift. I am keen to progress this research area in the broader ML community. To this end I actively foster cross-disciplinary international collaborations and have organized the `Shifts Challenge' on distributional shift at NeurIPS.

Publications

  • On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay

    OptimizationMachine learning theory
    Ekaterina Lobacheva
    Maxim Kodryan
    Nadezhda Chirkova
    Andrey Malinin
    Dmitry Vetrov
    NeurIPS,
    2021

    Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surprising periodic behavior of optimization dynamics: the training process regularly exhibits destabilizations that, however, do not lead to complete divergence but cause a new period of training. We rigorously investigate the mechanism underlying the discovered periodic behavior from both empirical and theoretical points of view and analyze the conditions in which it occurs in practice. We also demonstrate that periodic behavior can be regarded as a generalization of two previously opposing perspectives on training with batch normalization and weight decay, namely the equilibrium presumption and the instability presumption.

  • Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets

    Natural language processing Machine translationSpeech processingComputer visionOptimizationDistributional shiftUncertainty estimation Probabilistic machine learning
    Max Ryabinin
    Andrey Malinin
    Mark Gales
    NeurIPS,
    2021

    Ensembles of machine learning models yield improved system performance as well as robust and interpretable uncertainty estimates; however, their inference costs can be prohibitively high. Ensemble Distribution Distillation (EnD^2) is an approach that allows a single model to efficiently capture both the predictive performance and uncertainty estimates of an ensemble. For classification, this is achieved by training a Dirichlet distribution over the ensemble members' output distributions via the maximum likelihood criterion. Although theoretically principled, this work shows that the criterion exhibits poor convergence when applied to large-scale tasks where the number of classes is very high. Specifically, we show that for the Dirichlet log-likelihood criterion classes with low probability induce larger gradients than high-probability classes. Hence during training the model focuses on the distribution of the ensemble tail-class probabilities rather than the probability of the correct and closely related classes. We propose a new training objective which minimizes the reverse KL-divergence to a Proxy-Dirichlet target derived from the ensemble. This loss resolves the gradient issues of EnD^2, as we demonstrate both theoretically and empirically on the ImageNet, LibriSpeech, and WMT17 En-De datasets containing 1000, 5000, and 40,000 classes, respectively.

  • Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

    Machine translationNatural language processing Tabular dataDistributional shiftUncertainty estimation
    Andrey Malinin
    Neil Band
    Yarin Gal
    Mark J. F. Gales
    Alexander Ganshin
    German Chesnokov
    Alexey Noskov
    Andrey Ploskonosov
    Liudmila Prokhorenkova
    Ivan Provilkov
    Vatsal Raina
    Vyas Raina
    Denis Roginskiy
    Mariya Shmatova
    Panos Tigas
    Boris Yangel
    NeurIPS Benchmarks,
    2021

    Published at NeurIPS Datasets and Benchmarks Track.

    There has been significant research done on developing methods for improving robustness to distributional shift and uncertainty estimation. In contrast, only limited work has examined developing standard datasets and benchmarks for assessing these approaches. Additionally, most work on uncertainty estimation and robustness has developed new techniques based on small-scale regression or image classification tasks. However, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction. Thus, given the current state of the field, a standardized large-scale dataset of tasks across a range of modalities affected by distributional shifts is necessary. This will enable researchers to meaningfully evaluate the plethora of recently developed uncertainty quantification methods, as well as assessment criteria and state-of-the-art baselines. In this work, we propose the \emph{Shifts Dataset} for evaluation of uncertainty estimates and robustness to distributional shift. The dataset, which has been collected from industrial sources and services, is composed of three tasks, with each corresponding to a particular data modality: tabular weather prediction, machine translation, and self-driving car (SDC) vehicle motion prediction. All of these data modalities and tasks are affected by real, ‘in-the-wild’ distributional shifts and pose interesting challenges with respect to uncertainty estimation. In this work we provide a description of the dataset and baseline results for all tasks.

Posts