arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Kolyan Ray

27 papers · 2012–2025

2012: 12013: 02014: 12015: 22016: 12017: 12018: 32019: 22020: 32021: 22022: 42023: 42024: 22025: 1
2012peak 4 in 20222025

Frequent co-authors

Papers

showing 1–25 of 27 RSS

Variational Bayes for high-dimensional linear regression with sparse priors

Kolyan Ray, Botond Szabo

stat.ME · · cited 126× · arXiv ↗ · more like this ·

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference (CAVI) algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in…

Bayesian inverse problems with non-conjugate priors

Kolyan Ray

math.ST · · cited 89× · arXiv ↗ · more like this ·

We investigate the frequentist posterior contraction rate of nonparametric Bayesian procedures in linear inverse problems in both the mildly and severely ill-posed cases. A theorem is proved in a general Hilbert space setting under approximation-theoretic assumptions on the prior. The result is applied to non-conjugate priors, notably sieve and wavelet series priors, as well as in the conjugate setting. In the mildly ill-posed setting minimax optimal rates are obtained, with sieve priors being rate adaptive over Sobolev classes. In the severely ill-posed setting, oversmoothing the prior yields minimax rates. Previously established results in the conjugate setting are obtained using this method. Examples of applications include deconvolution, recovering the initial condition in the heat equation and the Radon transform.

Nonparametric statistical inference for drift vector fields of multi-dimensional diffusions

Richard Nickl, Kolyan Ray

math.ST · · cited 60× · arXiv ↗ · more like this ·

The problem of determining a periodic Lipschitz vector field $b=(b_1, \dots, b_d)$ from an observed trajectory of the solution $(X_t: 0 \le t \le T)$ of the multi-dimensional stochastic differential equation \begin{equation*} dX_t = b(X_t)dt + dW_t, \quad t \geq 0, \end{equation*} where $W_t$ is a standard $d$-dimensional Brownian motion, is considered. Convergence rates of a penalised least squares estimator, which equals the maximum a posteriori (MAP) estimate corresponding to a high-dimensional Gaussian product prior, are derived. These results are deduced from corresponding contraction rates for the associated posterior distributions. The rates obtained are optimal up to log-factors in $L^2$-loss in any dimension, and also for supremum norm loss when $d \le 4$. Further, when $d \le 3$, nonparametric Bernstein-von Mises theorems are proved for the posterior distributions of $b$. From…

Adaptive Bernstein-von Mises theorems in Gaussian white noise

Kolyan Ray

math.ST · · cited 58× · arXiv ↗ · more like this ·

We investigate Bernstein-von Mises theorems for adaptive nonparametric Bayesian procedures in the canonical Gaussian white noise model. We consider both a Hilbert space and multiscale setting with applications in $L^2$ and $L^\infty$ respectively. This provides a theoretical justification for plug-in procedures, for example the use of certain credible sets for sufficiently smooth linear functionals. We use this general approach to construct optimal frequentist confidence sets based on the posterior distribution. We also provide simulations to numerically illustrate our approach and obtain a visual representation of the geometries involved.

Spike and slab variational Bayes for high dimensional logistic regression

Kolyan Ray, Botond Szabo, Gabriel Clara

stat.ML · · cited 35× · arXiv ↗ · more like this ·

Variational Bayes (VB) is a popular scalable alternative to Markov chain Monte Carlo for Bayesian inference. We study a mean-field spike and slab VB approximation of widely used Bayesian model selection priors in sparse high-dimensional logistic regression. We provide non-asymptotic theoretical guarantees for the VB posterior in both $\ell_2$ and prediction loss for a sparse truth, giving optimal (minimax) convergence rates. Since the VB algorithm does not depend on the unknown truth to achieve optimality, our results shed light on effective prior choices. We confirm the improved performance of our VB algorithm over common sparse VB approaches in a numerical study.

Nonparametric Bayesian inference for reversible multi-dimensional diffusions

Matteo Giordano, Kolyan Ray

math.ST · · cited 27× · arXiv ↗ · more like this ·

We study nonparametric Bayesian models for reversible multi-dimensional diffusions with periodic drift. For continuous observation paths, reversibility is exploited to prove a general posterior contraction rate theorem for the drift gradient vector field under approximation-theoretic conditions on the induced prior for the invariant measure. The general theorem is applied to Gaussian priors and $p$-exponential priors, which are shown to converge to the truth at the minimax optimal rate over Sobolev smoothness classes in any dimension.

The Le Cam distance between density estimation, Poisson processes and Gaussian white noise

Kolyan Ray, Johannes Schmidt-Hieber

math.ST · · cited 12× · arXiv ↗ · more like this ·

It is well-known that density estimation on the unit interval is asymptotically equivalent to a Gaussian white noise experiment, provided the densities have Hölder smoothness larger than $1/2$ and are uniformly bounded away from zero. We derive matching lower and constructive upper bounds for the Le Cam deficiencies between these experiments, with explicit dependence on both the sample size and the size of the densities in the parameter space. As a consequence, we derive sharp conditions on how small the densities can be for asymptotic equivalence to hold. The related case of Poisson intensity estimation is also treated.

Semiparametric Bayesian causal inference

Kolyan Ray, Aad van der Vaart

math.ST · · cited 9× · arXiv ↗ · more like this ·

We develop a semiparametric Bayesian approach for estimating the mean response in a missing data model with binary outcomes and a nonparametrically modelled propensity score. Equivalently we estimate the causal effect of a treatment, correcting nonparametrically for confounding. We show that standard Gaussian process priors satisfy a semiparametric Bernstein-von Mises theorem under smoothness conditions. We further propose a novel propensity score-dependent prior that provides efficient inference under strictly weaker conditions. We also show that it is theoretically preferable to model the covariate distribution with a Dirichlet process or Bayesian bootstrap, rather than modelling the covariate density using a Gaussian process prior.

A Bayesian nonparametric approach to log-concave density estimation

Ester Mariucci, Kolyan Ray, Botond Szabo

math.ST · · cited 8× · arXiv ↗ · more like this ·

The estimation of a log-concave density on $\mathbb{R}$ is a canonical problem in the area of shape-constrained nonparametric inference. We present a Bayesian nonparametric approach to this problem based on an exponentiated Dirichlet process mixture prior and show that the posterior distribution converges to the log-concave truth at the (near-) minimax rate in Hellinger distance. Our proof proceeds by establishing a general contraction result based on the log-concave maximum likelihood estimator that prevents the need for further metric entropy calculations. We also present two computationally more feasible approximations and a more practical empirical Bayes approach, which are illustrated numerically via simulations.

Bayesian Nonparametric Inference in McKean-Vlasov models

Richard Nickl, Grigorios A. Pavliotis, Kolyan Ray

math.ST · · cited 7× · arXiv ↗ · more like this ·

We consider nonparametric statistical inference on a periodic interaction potential $W$ from noisy discrete space-time measurements of solutions $ρ=ρ_W$ of the nonlinear McKean-Vlasov equation, describing the probability density of the mean field limit of an interacting particle system. We show how Gaussian process priors assigned to $W$ give rise to posterior mean estimators that exhibit fast convergence rates for the implied estimated densities $\bar ρ$ towards $ρ_W$. We further show that if the initial condition $ϕ$ is not too smooth and satisfies a standard deconvolvability condition, then one can consistently infer Sobolev-regular potentials $W$ at convergence rates $N^{-θ}$ for appropriate $θ>0$, where $N$ is the number of measurements. The exponent $θ$ can be taken to approach $1/2$ as the regularity of $W$ increases corresponding to `near-parametric' models.

On the inability of Gaussian process regression to optimally learn compositional functions

Matteo Giordano, Kolyan Ray, Johannes Schmidt-Hieber

stat.ML · · cited 7× · arXiv ↗ · more like this ·

We rigorously prove that deep Gaussian process priors can outperform Gaussian process priors if the target function has a compositional structure. To this end, we study information-theoretic lower bounds for posterior contraction rates for Gaussian process regression in a continuous regression model. We show that if the true function is a generalized additive function, then the posterior based on any mean-zero Gaussian process can only recover the truth at a rate that is strictly slower than the minimax rate by a factor that is polynomially suboptimal in the sample size $n$.

Minimax theory for a class of non-linear statistical inverse problems

Kolyan Ray, Johannes Schmidt-Hieber

math.ST · · cited 7× · arXiv ↗ · more like this ·

We study a class of statistical inverse problems with non-linear pointwise operators motivated by concrete statistical applications. A two-step procedure is proposed, where the first step smoothes the data and inverts the non-linearity. This reduces the initial non-linear problem to a linear inverse problem with deterministic noise, which is then solved in a second step. The noise reduction step is based on wavelet thresholding and is shown to be minimax optimal (up to logarithmic factors) in a pointwise function-dependent sense. Our analysis is based on a modified notion of Hölder smoothness scales that are natural in this setting.

Variational Bayes for high-dimensional proportional hazards models with applications within gene expression

Michael Komodromos, Eric Aboagye, Marina Evangelou +2 more

stat.ME · · cited 6× · arXiv ↗ · more like this ·

Few Bayesian methods for analyzing high-dimensional sparse survival data provide scalable variable selection, effect estimation and uncertainty quantification. Such methods often either sacrifice uncertainty quantification by computing maximum a posteriori estimates, or quantify the uncertainty at high (unscalable) computational expense. We bridge this gap and develop an interpretable and scalable Bayesian proportional hazards model for prediction and variable selection, referred to as SVB. Our method, based on a mean-field variational approximation, overcomes the high computational cost of MCMC whilst retaining useful features, providing a posterior distribution for the parameters and offering a natural mechanism for variable selection via posterior inclusion probabilities. The performance of our proposed method is assessed via extensive simulations and compared against other…

A regularity class for the roots of non-negative functions

Kolyan Ray, Johannes Schmidt-Hieber

math.FA · · cited 6× · arXiv ↗ · more like this ·

We investigate the regularity of the positive roots of a non-negative function of one-variable. A modified Hölder space $\mathcal{F}^β$ is introduced such that if $f\in \mathcal{F}^β$ then $f^α\in C^{αβ}$. This provides sufficient conditions to overcome the usual limitation in the square root case ($α= 1/2$) for Hölder functions that $f^{1/2}$ need be no more than $C^1$ in general. We also derive bounds on the wavelet coefficients of $f^α$, which provide a finer understanding of its local regularity.

Debiased Bayesian inference for average treatment effects

Kolyan Ray, Botond Szabo

stat.ML · · cited 5× · arXiv ↗ · more like this ·

Bayesian approaches have become increasingly popular in causal inference problems due to their conceptual simplicity, excellent performance and in-built uncertainty quantification ('posterior credible sets'). We investigate Bayesian inference for average treatment effects from observational data, which is a challenging problem due to the missing counterfactuals and selection bias. Working in the standard potential outcomes framework, we propose a data-driven modification to an arbitrary (nonparametric) prior based on the propensity score that corrects for the first-order posterior bias, thereby improving performance. We illustrate our method for Gaussian process (GP) priors using (semi-)synthetic data. Our experiments demonstrate significant improvement in both estimation accuracy and uncertainty quantification compared to the unmodified GP, rendering our approach highly competitive…

Semiparametric inference using fractional posteriors

Alice L'Huillier, Luke Travis, Ismaël Castillo +1 more

math.ST · · cited 2× · arXiv ↗ · more like this ·

We establish a general Bernstein--von Mises theorem for approximately linear semiparametric functionals of fractional posterior distributions based on nonparametric priors. This is illustrated in a number of nonparametric settings and for different classes of prior distributions, including Gaussian process priors. We show that fractional posterior credible sets can provide reliable semiparametric uncertainty quantification, but have inflated size. To remedy this, we further propose a \textit{shifted-and-rescaled} fractional posterior set that is an efficient confidence set having optimal size under regularity conditions. As part of our proofs, we also refine existing contraction rate results for fractional posteriors by sharpening the dependence of the rate on the fractional exponent.

Nonparametric Bayesian estimation in a multidimensional diffusion model with high frequency data

Marc Hoffmann, Kolyan Ray

math.ST · · cited 2× · arXiv ↗ · more like this ·

We consider nonparametric Bayesian inference in a multidimensional diffusion model with reflecting boundary conditions based on discrete high-frequency observations. We prove a general posterior contraction rate theorem in $L^2$-loss, which is applied to Gaussian priors. The resulting posteriors, as well as their posterior means, are shown to converge to the ground truth at the minimax optimal rate over Hölder smoothness classes in any dimension. Of independent interest and as part of our proofs, we show that certain frequentist penalized least squares estimators are also minimax optimal.

On the Bernstein-von Mises theorem for the Dirichlet process

Kolyan Ray, Aad van der Vaart

math.ST · · cited 1× · arXiv ↗ · more like this ·

We establish that Laplace transforms of the posterior Dirichlet process converge to those of the limiting Brownian bridge process in a neighbourhood about zero, uniformly over Glivenko-Cantelli function classes. For real-valued random variables and functions of bounded variation, we strengthen this result to hold for all real numbers. This last result is proved via an explicit strong approximation coupling inequality.

Semiparametric Bernstein-von Mises theorems for reversible diffusions

Matteo Giordano, Kolyan Ray

math.ST · · cited 0× · arXiv ↗ · more like this ·

We establish a general semiparametric Bernstein-von Mises theorem for Bayesian nonparametric priors based on continuous observations in a periodic reversible multidimensional diffusion model. We consider a wide range of functionals satisfying an approximate linearization condition, including several nonlinear functionals of the invariant measure. Our result is applied to Gaussian and Besov-Laplace priors, showing these can perform efficient semiparametric inference and thus justifying the corresponding Bayesian approach to uncertainty quantification. Our theoretical results are illustrated via numerical simulations.

A variational Bayes approach to inference for low-dimensional parameters in high-dimensional linear regression

Ismaël Castillo, Alice L'Huillier, Kolyan Ray +1 more

stat.ML · · cited 0× · arXiv ↗ · more like this ·

We propose a scalable variational Bayes method for statistical inference for a single or pre-specified low-dimensional subset of the coordinates of a high-dimensional parameter in sparse linear regression. Our approach relies on assigning a mean-field approximation to the nuisance coordinates and carefully modelling the conditional distribution of the target given the nuisance. This requires only a preprocessing step and preserves the computational advantages of mean-field variational Bayes, while ensuring accurate and reliable inference for the target parameter, including for uncertainty quantification. We investigate the numerical performance of our algorithm, showing that it performs competitively with existing methods. We further establish accompanying theoretical guarantees for estimation and uncertainty quantification in the form of a Bernstein--von Mises theorem for a possibly…

Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior

Luke Travis, Kolyan Ray

math.ST · · cited 0× · arXiv ↗ · more like this ·

We study pointwise estimation and uncertainty quantification for a sparse variational Gaussian process method with eigenvector inducing variables. For a rescaled Brownian motion prior, we derive theoretical guarantees and limitations for the frequentist size and coverage of pointwise credible sets. For sufficiently many inducing variables, we precisely characterize the asymptotic frequentist coverage, deducing when credible sets from this variational method are conservative and when overconfident/misleading. We numerically illustrate the applicability of our results and discuss connections with other common Gaussian process priors.

Group Spike and Slab Variational Bayes

Michael Komodromos, Marina Evangelou, Sarah Filippi +1 more

stat.ME · · cited 0× · arXiv ↗ · more like this ·

We introduce Group Spike-and-slab Variational Bayes (GSVB), a scalable method for group sparse regression. A fast co-ordinate ascent variational inference (CAVI) algorithm is developed for several common model families including Gaussian, Binomial and Poisson. Theoretical guarantees for our proposed approach are provided by deriving contraction rates for the variational posterior in grouped linear regression. Through extensive numerical studies, we demonstrate that GSVB provides state-of-the-art performance, offering a computationally inexpensive substitute to MCMC, whilst performing comparably or better than existing MAP methods. Additionally, we analyze three real world datasets wherein we highlight the practical utility of our method, demonstrating that GSVB provides parsimonious models with excellent predictive performance, variable selection and uncertainty quantification.

Kolyan Ray and Botond Szabo's contribution to the Discussion of ‘Martingale Posterior Distributions’ by Fong, Holmes and Walker

Kolyan Ray, Botond Szabó

Artificial Intelligence · Journal of the Royal Statistical Society Series B (Statistical Methodology) · · cited 0× · DOI ↗ · more like this ·

We congratulate the authors for their thought-provoking article, which includes proposing a copula-based update for the predictive density and establishing its frequentist consistency under relatively mild assumptions. In this contribution, we further explore the frequentist properties of (Bayesian) predictive densities and illustrate through a simple example that, similarly to the posterior distribution, the predictive distribution can also be inconsistent. One therefore requires caution when using martingale posteriors, at least for the frequentist. Consider a modified version of Example 1 in the present paper, taken from Christensen (2009). Let Y1,…,Yn∼iidfθ⁠, where i.e. for rational parameter values θ, we replace the N(θ,1) Gaussian density by a Cauchy with location parameter θ. As in Example 1, we endow θ with a standard Gaussian prior, i.e. π(θ)=N(θ∣0,1)⁠. Since the likelihoods in…

Bayesian estimation in a multidimensional diffusion model with high frequency data

Marc Hoffmann, Kolyan Ray

Statistics and Probability · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 0× · OpenAlex ↗ · more like this ·

We consider nonparametric Bayesian inference in a multidimensional diffusion model with reflecting boundary conditions based on discrete high-frequency observations. We prove a general posterior contraction rate theorem in L 2-loss, which is applied to Gaussian priors. The resulting posteriors, as well as their posterior means, are shown to converge to the ground truth at the minimax optimal rate over H ölder smoothness classes in any dimension. Of independent interest and as part of our proofs, we show that certain frequentist penalized least squares estimators are also minimax optimal.

sparsevb: Spike-and-Slab Variational Bayes for Linear and Logistic Regression: R package version 0.1.0

Gabriel Clara, Botond Szabó, Kolyan Ray

Artificial Intelligence · University of Twente Research Information · · cited 0× · OpenAlex ↗ · more like this ·

Implements variational Bayesian algorithms to perform scalable variable selection for sparse, high-dimensional linear and logistic regression models. Features include a novel prioritized updating scheme, which uses a preliminary estimator of the variational means during initialization to generate an updating order prioritizing large, more relevant, coefficients. Sparsity is induced via spike-and-slab priors with either Laplace or Gaussian slabs. By default, the heavier-tailed Laplace density is used. Formal derivations of the algorithms and asymptotic consistency results may be found in Kolyan Ray and Botond Szabo (JASA 2020) and Kolyan Ray, Botond Szabo, and Gabriel Clara (NeurIPS 2020).