Ivan Rubachev

Yury Gorishniy

Y. Gorishniy, I. Rubachev, A. Babenko. “On embeddings for numerical features in tabular deep learning.” NeurIPS 2022.


Tabular data appears in many production systems. Recommender algorithms, delivery time estimation, fraud detection and other applications rely on processing tabular data. At Yandex Research, we actively contribute to the field of tabular machine learning.

In this post, we share exciting findings from our recent NeurIPS 2022 paper on deep learning and tabular data \[ref\]. We discuss how improved representations of numerical features can turn your vanilla MLPs (multilayer perceptrons) and Transformers into state-of-the-art tabular deep learning solutions that are competitive even with gradient-boosted decision trees, one of the strongest baselines in the world of tabular data problems!

Y. Gorishniy, I. Rubachev, V. Khrulkov, A. Babenko. “Revisiting deep learning models for tabular data.” NeurIPS 2021.

G. Somepalli et al. “SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training.” 


With Transformers taking over deep learning by storm, tabular data was no exception. Multiple works proposed Transformer-based architectures for tabular data, which outperformed MLP-like models \[ref\]\[ref\]. Compared to MLPs, the proposed Transformer architectures handle numerical features in a special way: they map scalar values of numerical features to high-dimensional embedding vectors, as shown in the picture below. Typically, this mapping is simple (e.g. a linear layer) and is not considered an important design choice. However, as we will see below, this mapping can significantly impact performance!

post-embeddings-for-numerical-features-1.png

large_post-embeddings-for-numerical-features-1.png

medium_post-embeddings-for-numerical-features-1.png

small_post-embeddings-for-numerical-features-1.png

thumbnail_post-embeddings-for-numerical-features-1.png

The crucial insights on the importance of input representation

M. Tancik et al. “Fourier features let networks learn high frequency functions in low dimensional domains.” NeurIPS 2020.



Apart from the above note on the transformers, we had one more important source of inspiration. In fact, proper input handling is one of the central topics in the field of implicit neural representations. In this field, neural networks are trained to map coordinate-like inputs (which are numerical features!) to different forms of content.

For example, the figure below illustrates how a neural network can fit an RGB image, which is equivalent to solving three tabular regression problems. In this specific example, the authors of \[ref\] demonstrated that a special input representation based on the Fourier Features allows solving the problem with dramatically better results compared to the raw scalar coordinates.

post-embeddings-for-numerical-features-2.png

large_post-embeddings-for-numerical-features-2.png

medium_post-embeddings-for-numerical-features-2.png

small_post-embeddings-for-numerical-features-2.png

thumbnail_post-embeddings-for-numerical-features-2.png

In fact, the above example showcases _the fundamental limitations of training MLPs with gradient-based optimization_. However, this is a whole different story. To learn more, we recommend the original paper \[ref4\] as a great entry point to the topic.

Overall, after making the above observations on the Transformers and implicit neural representations, we decided to explore different ways of encoding the numerical features before passing them to neural network backbones.

We start with “classical” machine learning techniques. We take inspiration from the one-hot encoding and adapt it for numerical features. Namely, for a given feature, we split its value range into bins and build the encoding as shown in the figure below (the example illustrates the encoding with four bins):

post-embeddings-for-numerical-features-3.png

large_post-embeddings-for-numerical-features-3.png

medium_post-embeddings-for-numerical-features-3.png

small_post-embeddings-for-numerical-features-3.png

thumbnail_post-embeddings-for-numerical-features-3.png

In the paper, we describe two ways of obtaining the bins and compare the scheme with the well-known feature binning technique.

Encoding with periodic activation functions

The second numerical embedding scheme is inspired by the success of periodic activations in the implicit neural representations. The specific embedding module is illustrated in the figure below. Importantly, we empirically observed that the initialization scale of the vector of “frequencies” (denoted as “c” in the figure) has a significant impact on the final performance.

post-embeddings-for-numerical-features-4.png

large_post-embeddings-for-numerical-features-4.png

medium_post-embeddings-for-numerical-features-4.png

small_post-embeddings-for-numerical-features-4.png

thumbnail_post-embeddings-for-numerical-features-4.png

While embedding modules described above enabled us to achieve good results, we observed that adding a linear layer with the ReLU non-linearity on top of them usually improves the results even further.

We apply the embeddings to both MLPs and Transformers. We pass embeddings to Transformers in a usual manner as a set of vectors. When passing embeddings to MLP, we simply concatenate them as shown below:

post-embedding-for-numerical-features-5.png

large_post-embedding-for-numerical-features-5.png

medium_post-embedding-for-numerical-features-5.png

small_post-embedding-for-numerical-features-5.png

thumbnail_post-embedding-for-numerical-features-5.png

The results are inspiring, to say the least!  _On a tabular benchmark, where baseline neural networks lag behind well-tuned gradient-boosted decision trees,_ the addition of embeddings to numerical features makes neural networks competitive with the tree-based models!

The second significant result is that MLP equipped with embeddings for numerical features performs on par with the heavy Transformer architecture.

We found that embeddings for numerical features can provide significant performance improvements for tabular deep models, and make vanilla MLPs competitive with heavy Transformer models.

That said, we only scratched the surface here. We certainly did not cover all possible embeddings, and choosing the best hyperparameters may still be an issue. Nevertheless, we are excited to share these results and continue to work on understanding and building upon them.

The source code for reproducing our experiments is available on [GitHub](https://github.com/Yura52/tabular-dl-num-embeddings).

Tabular data involves two-dimensional tables with objects (rows) and features (columns), which are used in numerous applied tasks such as classification, regression, ranking and many others.

Tabular data

In this post, we share exciting findings from our recent NeurIPS 2022 paper on deep learning and tabular data. We discuss how improved representations of numerical features can turn your vanilla MLPs (multilayer perceptrons) and Transformers into state-of-the-art tabular deep learning solutions that are competitive even with gradient-boosted decision trees, one of the strongest baselines in the world of tabular data problems!

Embeddings for numerical features in tabular deep learning

The ubiquitous transformers

The crucial insights on the importance of input representation

Piecewise linear encodings

Encoding with periodic activation functions

Using numerical feature embeddings

Results

Summary

References