Four papers by the Yandex Research team and our collaborators have been accepted for publication at the International Conference on Machine Learning (ICML 2023).
TabDDPM: Modelling Tabular Data with Diffusion Models by Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, Artem Babenko

Denoising diffusion probabilistic models are currently becoming the leading paradigm of generative modeling for many data modalities. In this work, we investigate if the framework of diffusion models can be advantageous for general tabular problems, where datapoints are represented by vectors of heterogeneous features. We introduce TabDDPM — a diffusion model that can be universally applied to any tabular dataset and handles any feature type. We extensively evaluate TabDDPM on a wide set of benchmarks and demonstrate its superiority over existing GAN/VAE alternatives, consistent with the advantage of diffusion models in other fields. Additionally, we show that TabDDPM is eligible for privacy-oriented setups, where the original datapoints cannot be publicly shared. Our approach is also compared with a simple SMOTE baseline, which, despite its simplicity, is hard to beat.

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient by Max Ryabinin, Tim Dettmers, Michael Diskin, Alexander Borzunov

We present a new approach for training large neural networks that is cost-efficient and accessible to researchers with limited resources. This approach relies on the finding that pipeline-parallel training larger models is less communication-intensive relative to their compute costs. Our proposed method, SWARM parallelism, enables the training of large models on heterogeneous and unreliable devices, such as preemptible instances or volunteer servers. This method achieves robustness to network failures by creating randomized pipelines between nodes that are rebalanced in case of a failure. We show that SWARM outperforms existing model-parallel algorithms in these conditions and can train a large Transformer language model with 1B shared parameters on preemptible T4 GPUs with less than 200Mb/s network speed.

High-throughput Generative Inference of Large Language Models with a Single GPU by Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang

Large language models (LLMs) typically require powerful hardware because of their high computational and memory requirements. Our paper introduces FlexGen, a new offloaded generation engine that allows these models to run efficiently on a single commodity GPU. FlexGen adapts to resource constraints by aggregating memory and computation from the GPU, CPU, and disk. It uses a linear programming optimizer to find the best way to store and access data, and compresses model weights and attention cache to just 4 bits without sacrificing accuracy. FlexGen outperforms state-of-the-art solutions for inference with offloading, achieving a generation throughput of 1 token/s on a single 16GB GPU for OPT-175B. With FlexGen, models with 30B parameters can be evaluated with one 16GB GPU on a wide range of HELM benchmark scenarios in less than 24 hours.


Which Tricks are Important for Learning to Rank? by Ivan Lyzhin, Aleksei Ustimenko, Andrey Gulin, Liudmila Prokhorenkova

The most well-known learning-to-rank algorithm is LambdaMART. LambdaMART is based on gradient-boosted decision trees (GBDT) and was proposed more than a decade ago. Since then, several other GBDT-based ranking methods have been proposed. In this paper, we thoroughly analyze these methods in a unified setup and show which algorithmic details are important for GBDT-based ranking. In addition to LambdaMART, we cover another long-known algorithm called YetiRank and the recently proposed StochasticRank. We empirically show that YetiRank outperforms its competitors in most cases. We also propose a simple improvement of the YetiRank approach that allows for optimizing specific ranking loss functions. As a result, we gain insights into learning-to-rank techniques and obtain a new state-of-the-art algorithm.

Papers accepted to ICML 2023

Four papers by the Yandex Research team have been accepted for publication at the International Conference on Learning Representations (ICLR 2023).

Papers accepted to ICLR 2023

Two papers have been accepted for publication at the Conference on Empirical Methods in Natural Language Processing (EMNLP 2022).

Papers accepted to EMNLP 2022

At Yandex Research, we advance progress in science and develop technologies for a better tomorrow.

large_join.jpg

medium_join.jpg

small_join.jpg

thumbnail_join.jpg

join.jpg

Join our research team and get an opportunity to engage in world-class applied and theoretical research, gain experience working on high-impact projects and publish at top-tier academic conferences. You will contribute to high-tech services like computer vision technologies, dialogue systems, neural machine translation and self-driving cars. 

Work with us

Yandex Research team regularly contributes to the computer vision research community, mostly in the field of image retrieval and generative modelling.

Computer vision

Language is one of the key forms of communication. We study methods of language representation and understanding to simplify human-computer interactions.

Natural language processing 

Today, training most powerful models often takes significant resources. Our research aims to make large-scale training more efficient and accessible to the entire machine learning community.

Large-scale machine learning

We study various aspects related to theoretical understanding of ML models and algorithms.

Machine learning theory

Graphs are a natural way to represent data from various domains such as social networks, molecules, text, code, etc. We develop and analyze algorithms for graph-structured data.

Graph machine learning

Probabilistic machine learning describes methods which enable reasoning and inference over unknown quantities. Commonly used in generative modelling, regression and uncertainty quantification.

Probabilistic machine learning

icon_post_20.svg

We have proposed a benchmark of five diverse heterophilous graphs that come from different domains and exhibit a variety of structural properties. Our benchmark includes a word dependency graph roman-empire, a product co-purchasing network amazon-ratings, a synthetic graph emulating the minesweeper game minesweeper, a crowdsourcing platform worker network tolokers, and a question-answering website interaction network questions.

Introducing new heterophilous graph datasets

icon_post_27.svg

In this post, we share exciting findings from our recent NeurIPS 2022 paper on deep learning and tabular data. We discuss how improved representations of numerical features can turn your vanilla MLPs (multilayer perceptrons) and Transformers into state-of-the-art tabular deep learning solutions that are competitive even with gradient-boosted decision trees, one of the strongest baselines in the world of tabular data problems!

Follow the cutting-edge research in computer science

Blog

Publications

A Critical Look at the Evaluation of GNNs under Heterophily: Are We Really Making Progress?

A View of Mini-batch SGD via Generating Functions: Conditions of Convergence, Phase Transitions, Benefit from Negative Momenta

Gradient Boosting Performs Gaussian Process Inference

Understanding DDPM Latent Codes Through Optimal Transport