arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Nicholas J. Foti

29 papers · 2012–2026

2012: 42013: 12014: 22015: 22016: 32017: 52018: 42019: 12020: 32021: 22022: 02023: 02024: 02025: 12026: 1
2012peak 5 in 20172026

Frequent co-authors

Papers

showing 1–25 of 29 RSS

Quantitative patterns of stylistic influence in the evolution of literature

James M. Hughes, Nicholas J. Foti, David C. Krakauer +1 more

Artificial Intelligence · Proceedings of the National Academy of Sciences · · cited 151× · DOI ↗ · more like this ·

Literature is a form of expression whose temporal structure, both in content and style, provides a historical record of the evolution of culture. In this work we take on a quantitative analysis of literary style and conduct the first large-scale temporal stylometric study of literature by using the vast holdings in the Project Gutenberg Digital Library corpus. We find temporal stylistic localization among authors through the analysis of the similarity structure in feature vectors derived from content-free word usage, nonhomogeneous decay rates of stylistic influence, and an accelerating rate of decay of influence among modern authors. Within a given time period we also find evidence for stylistic coherence with a given literary topic, such that writers in different fields adopt different literary styles. This study gives quantitative support to the notion of a literary "style of a time"…

Variational Boosting: Iteratively Refining Posterior Approximations

Andrew C. Miller, Nicholas Foti, Ryan P. Adams

stat.ML · · cited 134× · arXiv ↗ · more like this ·

We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trade computation time for accuracy. We show how to expand the variational approximating class by incorporating additional covariance structure and by introducing new components to form a mixture. We apply variational boosting to synthetic and real statistical models, and show that resulting posterior inferences compare favorably to existing posterior approximation algorithms in both accuracy and efficiency.

Neural Granger Causality

Alex Tank, Ian Covert, Nicholas Foti +2 more

stat.ML · · cited 107× · arXiv ↗ · more like this ·

While most classical approaches to Granger causality detection assume linear dynamics, many interactions in real-world applications, like neuroscience and genomics, are inherently nonlinear. In these cases, using linear models may lead to inconsistent estimation of Granger causal interactions. We propose a class of nonlinear methods by applying structured multilayer perceptrons (MLPs) or recurrent neural networks (RNNs) combined with sparsity-inducing penalties on the weights. By encouraging specific sets of weights to be zero--in particular, through the use of convex group-lasso penalties--we can extract the Granger causal structure. To further contrast with traditional approaches, our framework naturally enables us to efficiently capture long-range dependencies between series either via our RNNs or through an automatic lag selection in the MLP. We show that our neural Granger…

Stochastic Variational Inference for Hidden Markov Models

Nicholas J. Foti, Jason Xu, Dillon Laird +1 more

stat.ML · · cited 99× · arXiv ↗ · more like this ·

Variational inference algorithms have proven successful for Bayesian analysis in large data settings, with recent advances using stochastic variational inference (SVI). However, such methods have largely been studied in independent or exchangeable data settings. We develop an SVI algorithm to learn the parameters of hidden Markov models (HMMs) in a time-dependent data setting. The challenge in applying stochastic optimization in this setting arises from dependencies in the chain, which must be broken to consider minibatches of observations. We propose an algorithm that harnesses the memory decay of the chain to adaptively bound errors arising from edge effects. We demonstrate the effectiveness of our algorithm on synthetic experiments and a large genomics dataset where a batch algorithm is computationally infeasible.

Reducing Reparameterization Gradient Variance

Andrew C. Miller, Nicholas J. Foti, Alexander D'Amour +1 more

stat.ML · · cited 91× · arXiv ↗ · more like this ·

Optimization with noisy gradients has become ubiquitous in statistics and machine learning. Reparameterization gradients, or gradient estimates computed via the "reparameterization trick," represent a class of noisy gradients often used in Monte Carlo variational inference (MCVI). However, when these gradient estimators are too noisy, the optimization procedure can be slow or fail to converge. One way to reduce noise is to use more samples for the gradient estimate, but this can be computationally expensive. Instead, we view the noisy gradient as a random variable, and form an inexpensive approximation of the generating procedure for the gradient sample. This approximation has high correlation with the noisy gradient by construction, making it a useful control variate for variance reduction. We demonstrate our approach on non-conjugate multi-level hierarchical models and a Bayesian…

A survey of non-exchangeable priors for Bayesian nonparametric models

Nicholas J. Foti, Sinead Williamson

stat.ML · · cited 50× · arXiv ↗ · more like this ·

Dependent nonparametric processes extend distributions over measures, such as the Dirichlet process and the beta process, to give distributions over collections of measures, typically indexed by values in some covariate space. Such models are appropriate priors when exchangeability assumptions do not hold, and instead we want our model to vary fluidly with some set of covariates. Since the concept of dependent nonparametric processes was formalized by MacEachern [1], there have been a number of models proposed and used in the statistics and machine learning literatures. Many of these models exhibit underlying similarities, an understanding of which, we hope, will help in selecting an appropriate prior, developing new models, and leveraging inference techniques.

Mobility trends provide a leading indicator of changes in SARS-CoV-2 transmission

Andrew C. Miller, Nicholas J. Foti, Joseph A. Lewnard +3 more

Modeling and Simulation · medRxiv · · cited 42× · DOI ↗ · more like this ·

Determining the impact of non-pharmaceutical interventions on transmission of the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is paramount for the design and deployment of effective public health policies. Incorporating Apple Maps mobility data into an epidemiological model of daily deaths and hospitalizations allowed us to estimate an explicit relationship between human mobility and transmission in the United States. We find that reduced mobility explains a large decrease in the effective reproductive number ( R E ) attained by April 1st and further identify state-to-state variation in the inferred transmission-mobility relationship. These findings indicate that simply relaxing stay-at-home orders can rapidly lead to outbreaks exceeding the scale of transmission that has occurred to date. Our findings provide quantitative guidance on the impact policies must…

Stochastic Gradient MCMC Methods for Hidden Markov Models

Yi-An Ma, Nicholas J. Foti, Emily B. Fox

stat.ML · · cited 35× · arXiv ↗ · more like this ·

Stochastic gradient MCMC (SG-MCMC) algorithms have proven useful in scaling Bayesian inference to large datasets under an assumption of i.i.d data. We instead develop an SG-MCMC algorithm to learn the parameters of hidden Markov models (HMMs) for time-dependent data. There are two challenges to applying SG-MCMC in this setting: The latent discrete states, and needing to break dependencies when considering minibatches. We consider a marginal likelihood representation of the HMM and propose an algorithm that harnesses the inherent memory decay of the process. We demonstrate the effectiveness of our algorithm on synthetic experiments and an ion channel recording data, with runtimes significantly outperforming batch MCMC.

It’s complicated: characterizing the time-varying relationship between cell phone mobility and COVID-19 spread in the US

Sean Jewell, Joseph Futoma, Lauren A. Hannah +3 more

Modeling and Simulation · medRxiv · · cited 32× · DOI ↗ · more like this ·

Abstract Restricting in-person interactions is an important technique for limiting the spread of severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2). Although early research found strong associations between cell phone mobility and infection spread during the initial outbreaks in the United States, it is unclear whether this relationship persists across locations and time. We propose an interpretable statistical model to identify spatiotemporal variation in the association between mobility and infection rates. Using one year of US county-level data, we found that sharp drops in mobility often coincided with declining infection rates in the most populous counties in spring 2020. However, the association varied considerably in other locations and across time. Our findings are sensitive to model flexibility, as more restrictive models average over local effects and mask much of…

Statistical deconvolution for inference of infection time series

Andrew C. Miller, Lauren A. Hannah, Joseph Futoma +6 more

Modeling and Simulation · medRxiv · · cited 30× · DOI ↗ · more like this ·

Abstract Accurate measurement of daily infection incidence is crucial to epidemic response. However, delays in symptom onset, testing, and reporting obscure the dynamics of transmission, necessitating methods to remove the effects of stochastic delays from observed data. Existing estimators can be sensitive to model misspecification and censored observations; many analysts have instead used methods that exhibit strong bias or do not account for delays. We develop an estimator with a regularization scheme to cope with these sources of noise, which we term the Robust Incidence Deconvolution Estimator (RIDE). We validate RIDE on synthetic data, comparing accuracy and stability to existing approaches. We then use RIDE to study COVID-19 records in the United States, and find evidence that infection estimates from reported cases can be more informative than estimates from mortality data. To…

The Cultural Evolution of National Constitutions

Daniel N. Rockmore, Chen Fang, Nicholas J. Foti +2 more

cs.SI · · cited 30× · arXiv ↗ · more like this ·

We explore how ideas from infectious disease and genetics can be used to uncover patterns of cultural inheritance and innovation in a corpus of 591 national constitutions spanning 1789 - 2008. Legal "Ideas" are encoded as "topics" - words statistically linked in documents - derived from topic modeling the corpus of constitutions. Using these topics we derive a diffusion network for borrowing from ancestral constitutions back to the US Constitution of 1789 and reveal that constitutions are complex cultural recombinants. We find systematic variation in patterns of borrowing from ancestral texts and "biological"-like behavior in patterns of inheritance with the distribution of "offspring" arising through a bounded preferential-attachment process. This process leads to a small number of highly innovative (influential) constitutions some of which have yet to have been identified as so in the…

An Interpretable and Sparse Neural Network Model for Nonlinear Granger Causality Discovery

Alex Tank, Ian Cover, Nicholas J. Foti +2 more

stat.ML · · cited 26× · arXiv ↗ · more like this ·

While most classical approaches to Granger causality detection repose upon linear time series assumptions, many interactions in neuroscience and economics applications are nonlinear. We develop an approach to nonlinear Granger causality detection using multilayer perceptrons where the input to the network is the past time lags of all series and the output is the future value of a single series. A sufficient condition for Granger non-causality in this setting is that all of the outgoing weights of the input data, the past lags of a series, to the first hidden layer are zero. For estimation, we utilize a group lasso penalty to shrink groups of input weights to zero. We also propose a hierarchical penalty for simultaneous Granger causality and lag estimation. We validate our approach on simulated data from both a sparse linear autoregressive model and the sparse and nonlinear Lorenz-96…

Bayesian Structure Learning for Stationary Time Series

Alex Tank, Nicholas Foti, Emily Fox

stat.ME · · cited 26× · arXiv ↗ · more like this ·

While much work has explored probabilistic graphical models for independent data, less attention has been paid to time series. The goal in this setting is to determine conditional independence relations between entire time series, which for stationary series, are encoded by zeros in the inverse spectral density matrix. We take a Bayesian approach to structure learning, placing priors on (i) the graph structure and (ii) spectral matrices given the graph. We leverage a Whittle likelihood approximation and define a conjugate prior---the hyper complex inverse Wishart---on the complex-valued and graph-constrained spectral matrices. Due to conjugacy, we can analytically marginalize the spectral matrices and obtain a closed-form marginal likelihood of the time series given a graph. Importantly, our analytic marginal likelihood allows us to avoid inference of the complex spectral matrices…

Breiman's two cultures: You don't have to choose sides

Andrew C. Miller, Nicholas J. Foti, Emily B. Fox

stat.ML · · cited 12× · arXiv ↗ · more like this ·

Breiman's classic paper casts data analysis as a choice between two cultures: data modelers and algorithmic modelers. Stated broadly, data modelers use simple, interpretable models with well-understood theoretical properties to analyze data. Algorithmic modelers prioritize predictive accuracy and use more flexible function approximations to analyze data. This dichotomy overlooks a third set of models $-$ mechanistic models derived from scientific theories (e.g., ODE/SDE simulators). Mechanistic models encode application-specific scientific knowledge about the data. And while these categories represent extreme points in model space, modern computational and algorithmic tools enable us to interpolate between these points, producing flexible, interpretable, and scientifically-informed hybrids that can enjoy accurate and robust predictions, and resolve issues with data analysis that Breiman…

Slice sampling normalized kernel-weighted completely random measure mixture models

Nicholas J. Foti, Sinead A. Williamson

Artificial Intelligence · · cited 11× · OpenAlex ↗ · more like this ·

A number of dependent nonparametric processes have been proposed to model non-stationary data with unknown latent dimensionality. However, the inference algorithms are often slow and unwieldy, and are in general highly specific to a given model formulation. In this paper, we describe a large class of dependent nonparametric processes, including several existing models, and present a slice sampler that allows efficient inference across this class of models.

Interpretable VAEs for nonlinear group factor analysis

Samuel Ainsworth, Nicholas Foti, Adrian KC Lee +1 more

cs.LG · · cited 9× · arXiv ↗ · more like this ·

Deep generative models have recently yielded encouraging results in producing subjectively realistic samples of complex data. Far less attention has been paid to making these generative models interpretable. In many scenarios, ranging from scientific applications to finance, the observed variables have a natural grouping. It is often of interest to understand systems of interaction amongst these groups, and latent factor models (LFMs) are an attractive approach. However, traditional LFMs are limited by assuming a linear correlation structure. We present an output interpretable VAE (oi-VAE) for grouped data that models complex, nonlinear latent-to-observed relationships. We combine a structured VAE comprised of group-specific generators with a sparsity-inducing prior. We demonstrate that oi-VAE yields meaningful notions of interpretability in the analysis of motion capture and MEG data.…

A unifying representation for a class of dependent random measures

Nicholas J. Foti, Joseph D. Futoma, Daniel N. Rockmore +1 more

stat.ML · · cited 9× · arXiv ↗ · more like this ·

We present a general construction for dependent random measures based on thinning Poisson processes on an augmented space. The framework is not restricted to dependent versions of a specific nonparametric model, but can be applied to all models that can be represented using completely random measures. Several existing dependent random measures can be seen as specific cases of this framework. Interesting properties of the resulting measures are derived and the efficacy of the framework is demonstrated by constructing a covariate-dependent latent feature model and topic model that obtain superior predictive performance.

Streaming Variational Inference for Bayesian Nonparametric Mixture Models

Alex Tank, Nicholas J. Foti, Emily B. Fox

stat.ML · · cited 8× · arXiv ↗ · more like this ·

In theory, Bayesian nonparametric (BNP) models are well suited to streaming data scenarios due to their ability to adapt model complexity with the observed data. Unfortunately, such benefits have not been fully realized in practice; existing inference algorithms are either not applicable to streaming applications or not extensible to BNP models. For the special case of Dirichlet processes, streaming inference has been considered. However, there is growing interest in more flexible BNP models building on the class of normalized random measures (NRMs). We work within this general framework and present a streaming variational inference algorithm for NRM mixture models. Our algorithm is based on assumed density filtering (ADF), leading straightforwardly to expectation propagation (EP) for large-scale batch inference as well. We demonstrate the efficacy of the algorithm on clustering…

Disentangled VAE Representations for Multi-Aspect and Missing Data

Samuel K. Ainsworth, Nicholas J. Foti, Emily B. Fox

cs.LG · · cited 7× · arXiv ↗ · more like this ·

Many problems in machine learning and related application areas are fundamentally variants of conditional modeling and sampling across multi-aspect data, either multi-view, multi-modal, or simply multi-group. For example, sampling from the distribution of English sentences conditioned on a given French sentence or sampling audio waveforms conditioned on a given piece of text. Central to many of these problems is the issue of missing data: we can observe many English, French, or German sentences individually but only occasionally do we have data for a sentence pair. Motivated by these applications and inspired by recent progress in variational autoencoders for grouped data, we develop factVAE, a deep generative model capable of handling multi-aspect data, robust to missing observations, and with a prior that encourages disentanglement between the groups and the latent dimensions. The…

Adaptively Truncating Backpropagation Through Time to Control Gradient Bias

Christopher Aicher, Nicholas J. Foti, Emily B. Fox

cs.LG · · cited 2× · arXiv ↗ · more like this ·

Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge if the truncation length is too small, or will converge slowly if it is too large. We propose an adaptive TBPTT scheme that converts the problem from choosing a temporal lag to one of choosing a tolerable amount of gradient bias. For many realistic RNNs, the TBPTT gradients decay geometrically in expectation for large lags; under this condition, we can control the bias by varying the truncation length adaptively. For RNNs with smooth activation functions, we prove that this bias controls the convergence rate of SGD with biased gradients for our non-convex loss. Using this…

Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions

Eray Erturk, Fahad Kamran, Salar Abbaspourazad +6 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Wearable devices record physiological and behavioral signals that can improve health predictions. While foundation models are increasingly used for such predictions, they have been primarily applied to low-level sensor data, despite behavioral data often being more informative due to their alignment with physiologically relevant timescales and quantities. We develop foundation models of such behavioral signals using over 2.5B hours of wearable data from 162K individuals, systematically optimizing architectures and tokenization strategies for this unique dataset. Evaluated on 57 health-related tasks, our model shows strong performance across diverse real-world applications including individual-level classification and time-varying health state prediction. The model excels in behavior-driven tasks like sleep prediction, and improves further when combined with representations of raw sensor…

Stochastic Gradient MCMC for State Space Models

Christopher Aicher, Yi-An Ma, Nicholas J. Foti +1 more

stat.ML · · cited 0× · arXiv ↗ · more like this ·

State space models (SSMs) are a flexible approach to modeling complex time series. However, inference in SSMs is often computationally prohibitive for long time series. Stochastic gradient MCMC (SGMCMC) is a popular method for scalable Bayesian inference for large independent data. Unfortunately when applied to dependent data, such as in SSMs, SGMCMC's stochastic gradient estimates are biased as they break crucial temporal dependencies. To alleviate this, we propose stochastic gradient estimators that control this bias by performing additional computation in a `buffer' to reduce breaking dependencies. Furthermore, we derive error bounds for this bias and show a geometric decay under mild conditions. Using these estimators, we develop novel SGMCMC samplers for discrete, continuous and mixed-type SSMs with analytic message passing. Our experiments on real and synthetic data demonstrate…

Supplemental Information: Streaming Variational Inference for Bayesian Nonparametric Mixture Models

Alex Tank, Nicholas J. Foti, Emily B. Fox

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

This section motivates the use of the mean field update for the global variables, given in Eq. (19) of the main text, as an approximation to the optimal update for ADF. The presentation adapts that of [1] for ADF-NRM. Let p̂(θ, z1:n|x1:n) ∝ p(xn|θ, zn)p(zn|z1:n)q̂(θ, z1:n−1) denote the approximate posterior under the past variational updates after adding in the nth observation. The optimal q(θk) under ADF is given by the marginal distribution of p̂: q̂n(θk) ∝ z1:n p̂(θk|θ\\k, z1:n, x1:n)p̂(θ\\k|z1:n, x1:n)p̂(z1:n|x1:n)dθ\\k. (1) Both the sums and integrals are intractable so we use the approximations: p̂(θ\\k|z1:n, x1:n) ≈ q̂n(θ\\k) and p̂(z1:n|x1:n) ≈ q̂n(z1:n) = ∏n i=1 q̂n(zi) which yields: q̂n(θk) z1:n p̂(θk|θ\\k, z1:n, x1:n)q̂n(θ\\k)q̂n(z1:n)dθ\\k (2)

Streaming Variational Inference for Bayesian Nonparametric Mixture Models. arXiv.org

Alex Tank, Nicholas J. Foti, Emily B. Fox

Artificial Intelligence · · cited 0× · OpenAlex ↗ · more like this ·

In theory, Bayesian nonparametric (BNP) models are well suited to streaming data sce-narios due to their ability to adapt model complexity with the observed data. Unfor-tunately, such benefits have not been fully realized in practice; existing inference algo-rithms are either not applicable to stream-ing applications or not extensible to BNP models. For the special case of Dirichlet processes, streaming inference has been con-sidered. However, there is growing inter-est in more flexible BNP models building on the class of normalized random measures (NRMs). We work within this general frame-work and present a streaming variational in-ference algorithm for NRM mixture models. Our algorithm is based on assumed density filtering (ADF), leading straightforwardly to expectation propagation (EP) for large-scale batch inference as well. We demonstrate the efficacy of the algorithm on clustering…