This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem. Thus, we obtain the convergence to a Gaussian Process' posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance. We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection.

Aleksei Ustimenko

Artem Beliakov

Liudmila Prokhorenkova

We study various aspects related to theoretical understanding of ML models and algorithms.

Machine learning theory

Uncertainty estimation enables detecting when ML models make mistakes. This is of critical importance in high risk machine learning applications, such as autonomous vehicle and medical ML.

Uncertainty estimation 

Gradient boosting iteratively combines weak learners (usually decision trees) to create a stronger model. It achieves state-of-the-art results on tabular data with heterogeneous features.

Gradient Boosting Performs Gaussian Process Inference