arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Scott W. Linderman

61 papers · 2014–2026

2014: 32015: 22016: 42017: 32018: 52019: 22020: 32021: 42022: 52023: 62024: 52025: 122026: 7
2014peak 12 in 20252026

Frequent co-authors

Papers

showing 1–25 of 61 RSS

Simplified State Space Layers for Sequence Modeling

Jimmy T. H. Smith, Andrew Warrington, Scott W. Linderman

cs.LG · · cited 1,173× · arXiv ↗ · more like this ·

Models using structured state space sequence (S4) layers have achieved state-of-the-art performance on long-range sequence modeling tasks. An S4 layer combines linear state space models (SSMs), the HiPPO framework, and deep learning to achieve high performance. We build on the design of the S4 layer and introduce a new state space layer, the S5 layer. Whereas an S4 layer uses many independent single-input, single-output SSMs, the S5 layer uses one multi-input, multi-output SSM. We establish a connection between S5 and S4, and use this to develop the initialization and parameterization used by the S5 model. The result is a state space layer that can leverage efficient and widely implemented parallel scans, allowing S5 to match the computational efficiency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks. S5 averages 87.4% on the long…

Learning Latent Permutations with Gumbel-Sinkhorn Networks

Gonzalo Mena, David Belanger, Scott Linderman +1 more

stat.ML · · cited 354× · arXiv ↗ · more like this ·

Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data. Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable. In response, this paper introduces a collection of new methods for end-to-end learning in such models that approximate discrete maximum-weight matching using the continuous Sinkhorn operator. Sinkhorn iteration is attractive because it functions as a simple, easy-to-implement analog of the softmax operator. With this, we can define the Gumbel-Sinkhorn method, an extension of the Gumbel-Softmax method (Jang et al. 2016, Maddison2016 et al. 2016) to distributions over latent matchings. We demonstrate the effectiveness of our method by outperforming competitive baselines on a range of qualitatively different tasks…

Discovering Latent Network Structure in Point Process Data

Scott W. Linderman, Ryan P. Adams

stat.ML · · cited 297× · arXiv ↗ · more like this ·

Networks play a central role in modern data analysis, enabling us to reason about systems by studying the relationships between their parts. Most often in network analysis, the edges are given. However, in many systems it is difficult or impossible to measure the network directly. Examples of latent networks include economic interactions linking financial instruments and patterns of reciprocity in gang violence. In these cases, we are limited to noisy observations of events associated with each node. To enable analysis of these implicit networks, we develop a probabilistic model that combines mutually-exciting point processes with random graph models. We show how the Poisson superposition principle enables an elegant auxiliary variable formulation and a fully-Bayesian, parallel inference algorithm. We evaluate this new model empirically on several datasets.

Variational Sequential Monte Carlo

Christian A. Naesseth, Scott W. Linderman, Rajesh Ranganath +1 more

stat.ML · · cited 245× · arXiv ↗ · more like this ·

Many recent advances in large scale probabilistic inference rely on variational methods. The success of variational approaches depends on (i) formulating a flexible parametric family of distributions, and (ii) optimizing the parameters to find the member of this family that most closely approximates the exact posterior. In this paper we present a new approximating family of distributions, the variational sequential Monte Carlo (VSMC) family, and show how to optimize it in variational inference. VSMC melds variational inference (VI) and sequential Monte Carlo (SMC), providing practitioners with flexible, accurate, and powerful Bayesian inference. The VSMC family is a variational family that can approximate the posterior arbitrarily well, while still allowing for efficient optimization of its parameters. We demonstrate its utility on state space models, stochastic volatility models for…

Generalized Shape Metrics on Neural Representations

Alex H. Williams, Erin Kunz, Simon Kornblith +1 more

stat.ML · · cited 182× · arXiv ↗ · more like this ·

Understanding the operation of biological and artificial networks remains a difficult and important challenge. To identify general principles, researchers are increasingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools is now needed to identify how network-level covariates -- such as architecture, anatomical brain region, and model organism -- impact neural representations (hidden layer activations). Here, we provide a rigorous foundation for these analyses by defining a broad family of metric spaces that quantify representational dissimilarity. Using this framework we modify existing representational similarity measures based on canonical correlation analysis to satisfy the triangle inequality, formulate a novel metric that respects the inductive biases in convolutional layers…

Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms

Christian A. Naesseth, Francisco J. R. Ruiz, Scott W. Linderman +1 more

stat.ML · · cited 127× · arXiv ↗ · more like this ·

Variational inference using the reparameterization trick has enabled large-scale approximate Bayesian inference in complex probabilistic models, leveraging stochastic optimization to sidestep intractable expectations. The reparameterization trick is applicable when we can simulate a random variable by applying a differentiable deterministic function on an auxiliary random variable whose distribution is fixed. For many distributions of interest (such as the gamma or Dirichlet), simulation of random variables relies on acceptance-rejection sampling. The discontinuity introduced by the accept-reject step means that standard reparameterization tricks are not applicable. We propose a new method that lets us leverage reparameterization gradients even when variables are outputs of a acceptance-rejection sampling algorithm. Our approach enables reparameterization on a larger class of…

Dependent Multinomial Models Made Easy: Stick Breaking with the Pólya-Gamma Augmentation

Scott W. Linderman, Matthew J. Johnson, Ryan P. Adams

stat.ML · · cited 108× · arXiv ↗ · more like this ·

Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and year, and text documents are all commonly modeled with multinomial distributions. In all of these cases, we expect some form of dependency between the draws: the nucleotide at one position in the DNA strand may depend on the preceding nucleotides, children's names are highly correlated from year to year, and topics in text may be correlated and dynamic. These dependencies are not naturally captured by the typical Dirichlet-multinomial formulation. Here, we leverage a logistic stick-breaking representation and recent innovations in Pólya-gamma augmentation to reformulate the multinomial distribution in terms of latent variables with jointly Gaussian likelihoods, enabling…

Recurrent switching linear dynamical systems

Scott W. Linderman, Andrew C. Miller, Ryan P. Adams +3 more

stat.ML · · cited 98× · arXiv ↗ · more like this ·

Many natural systems, such as neurons firing in the brain or basketball teams traversing a court, give rise to time series data with complex, nonlinear dynamics. We can gain insight into these systems by decomposing the data into segments that are each explained by simpler dynamic units. Building on switching linear dynamical systems (SLDS), we present a new model class that not only discovers these dynamical units, but also explains how their switching behavior depends on observations or continuous latent states. These "recurrent" switching linear dynamical systems provide further insight by discovering the conditions under which each unit is deployed, something that traditional SLDS models fail to do. We leverage recent algorithmic advances in approximate inference to make Bayesian inference in these models easy, fast, and scalable.

Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modeling

Josue Nassar, Scott W. Linderman, Monica Bugallo +1 more

stat.ML · · cited 85× · arXiv ↗ · more like this ·

Many real-world systems studied are governed by complex, nonlinear dynamics. By modeling these dynamics, we can gain insight into how these systems work, make predictions about how they will behave, and develop strategies for controlling them. While there are many methods for modeling nonlinear dynamical systems, existing techniques face a trade off between offering interpretable descriptions and making accurate predictions. Here, we develop a class of models that aims to achieve both simultaneously, smoothly interpolating between simple descriptions and more complex, yet also more accurate models. Our probabilistic model achieves this multi-scale property through a hierarchy of locally linear dynamics that jointly approximate global nonlinear dynamics. We call it the tree-structured recurrent switching linear dynamical system. To fit this model, we present a fully-Bayesian sampling…

Scalable Bayesian Inference for Excitatory Point Process Networks

Scott W. Linderman, Ryan P. Adams

stat.ML · · cited 61× · arXiv ↗ · more like this ·

Networks capture our intuition about relationships in the world. They describe the friendships between Facebook users, interactions in financial markets, and synapses connecting neurons in the brain. These networks are richly structured with cliques of friends, sectors of stocks, and a smorgasbord of cell types that govern how neurons connect. Some networks, like social network friendships, can be directly observed, but in many cases we only have an indirect view of the network through the actions of its constituents and an understanding of how the network mediates that activity. In this work, we focus on the problem of latent network discovery in the case where the observable activity takes the form of a mutually-excitatory point process known as a Hawkes process. We build on previous work that has taken a Bayesian approach to this problem, specifying prior distributions over the…

Structure-Exploiting variational inference for recurrent switching linear dynamical systems

Scott W. Linderman, Matthew Johnson

Artificial Intelligence · · cited 32× · DOI ↗ · more like this ·

Many natural systems, such as neurons firing in the brain or basketball teams traversing a court, give rise to time series data with complex, nonlinear dynamics. We can gain insight into these systems by decomposing the data into segments that are each explained by simpler dynamic units. This is the motivation underlying the class of recurrent switching linear dynamical systems (rSLDS) [1], which build on the standard SLDS by introducing a model of how discrete transition probabilities depend on observations or continuous latent states. Previous work relied on Markov chain Monte Carlo algorithms and augmentation schemes for inference, but these methods only applied to a limited class of recurrent dependencies. Here we relax these constraints and consider recurrent dependencies specified by arbitrary parametric, nonlinear functions. We derive two structure-exploiting variational…

Poisson-Randomized Gamma Dynamical Systems

Aaron Schein, Scott W. Linderman, Mingyuan Zhou +2 more

stat.ML · · cited 26× · arXiv ↗ · more like this ·

This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness. The PRGDS is based on a new motif in Bayesian latent variable modeling, an alternating chain of discrete Poisson and continuous gamma latent states that is analytically convenient and computationally tractable. This motif yields closed-form complete conditionals for all variables by way of the Bessel distribution and a novel discrete distribution that we call the shifted confluent hypergeometric distribution. We draw connections to closely related models and compare the PRGDS to these models in studies of real-world count data sets of text, international events, and neural spike trains. We find that a sparse variant of the PRGDS, which allows the continuous gamma latent states to take values of…

Point process latent variable models of larval zebrafish behavior

Anuj Kumar Sharma, Robert E. Johnson, Florian Engert +1 more

Applied Mathematics · Neural Information Processing Systems · · cited 19× · OpenAlex ↗ · more like this ·

A fundamental goal of systems neuroscience is to understand how neural activity gives rise to natural behavior. In order to achieve this goal, we must first build comprehensive models that offer quantitative descriptions of behavior. We develop a new class of probabilistic models to tackle this challenge in the study of larval zebrafish, an important model organism for neuroscience. Larval zebrafish locomote via sequences of punctate swim bouts--brief flicks of the tail--which are naturally modeled as a marked point process. However, these sequences of swim bouts belie a set of discrete and continuous internal states, latent variables that are not captured by standard point process models. We incorporate these variables as latent marks of a point process and explore various models for their dynamics. To infer the latent variables and fit the parameters of this model, we develop an…

Mutually Regressive Point Processes

Ifigeneia Apostolopoulou, Scott W. Linderman, Kyle Miller +1 more

Applied Mathematics · Neural Information Processing Systems · · cited 17× · OpenAlex ↗ · more like this ·

Many real-world data represent sequences of interdependent events unfolding over time. They can be modeled naturally as realizations of a point process. Despite many potential applications, existing point process models are limited in their ability to capture complex patterns of interaction. Hawkes processes admit many efficient inference algorithms, but are limited to mutually excitatory effects. Non- linear Hawkes processes allow for more complex influence patterns, but for their estimation it is typically necessary to resort to discrete-time approximations that may yield poor generative models. In this paper, we introduce the first general class of Bayesian point process models extended with a nonlinear component that allows both excitatory and inhibitory relationships in continuous time. We derive a fully Bayesian inference algorithm for these processes using Polya-Gamma…

Point process models for sequence detection in high-dimensional neural spike trains

Alex H. Williams, Anthony Degleris, Yixin Wang +1 more

stat.ML · · cited 9× · arXiv ↗ · more like this ·

Sparse sequences of neural spikes are posited to underlie aspects of working memory, motor production, and learning. Discovering these sequences in an unsupervised manner is a longstanding problem in statistical neuroscience. Promising recent work utilized a convolutive nonnegative matrix factorization model to tackle this challenge. However, this model requires spike times to be discretized, utilizes a sub-optimal least-squares criterion, and does not provide uncertainty estimates for model predictions or estimated parameters. We address each of these shortcomings by developing a point process model that characterizes fine-scale sequences at the level of individual spikes and represents sequence occurrences as a small number of marked events in continuous time. This ultra-sparse representation of sequence events opens new possibilities for spike train modeling. For example, we…

Spatiotemporal Clustering with Neyman-Scott Processes via Connections to Bayesian Nonparametric Mixture Models

Yixin Wang, Anthony Degleris, Alex H. Williams +1 more

stat.ML · · cited 7× · arXiv ↗ · more like this ·

Neyman-Scott processes (NSPs) are point process models that generate clusters of points in time or space. They are natural models for a wide range of phenomena, ranging from neural spike trains to document streams. The clustering property is achieved via a doubly stochastic formulation: first, a set of latent events is drawn from a Poisson process; then, each latent event generates a set of observed data points according to another Poisson process. This construction is similar to Bayesian nonparametric mixture models like the Dirichlet process mixture model (DPMM) in that the number of latent events (i.e. clusters) is a random variable, but the point process formulation makes the NSP especially well suited to modeling spatiotemporal data. While many specialized algorithms have been developed for DPMMs, comparatively fewer works have focused on inference in NSPs. Here, we present novel…

Reparameterizing the Birkhoff Polytope for Variational Permutation Inference

Scott W. Linderman, Gonzalo E. Mena, Hal Cooper +2 more

stat.ML · · cited 7× · arXiv ↗ · more like this ·

Many matching, tracking, sorting, and ranking problems require probabilistic reasoning about possible permutations, a set that grows factorially with dimension. Combinatorial optimization algorithms may enable efficient point estimation, but fully Bayesian inference poses a severe challenge in this high-dimensional, discrete space. To surmount this challenge, we start with the usual step of relaxing a discrete set (here, of permutation matrices) to its convex hull, which here is the Birkhoff polytope: the set of all doubly-stochastic matrices. We then introduce two novel transformations: first, an invertible and differentiable stick-breaking procedure that maps unconstrained space to the Birkhoff polytope; second, a map that rounds points toward the vertices of the polytope. Both transformations include a temperature parameter that, in the limit, concentrates the densities on…

A Nonparametric Bayesian Approach to Uncovering Rat Hippocampal Population Codes During Spatial Navigation

Scott W. Linderman, Matthew J. Johnson, Matthew A. Wilson +1 more

stat.ML · · cited 6× · arXiv ↗ · more like this ·

Rodent hippocampal population codes represent important spatial information about the environment during navigation. Several computational methods have been developed to uncover the neural representation of spatial topology embedded in rodent hippocampal ensemble spike activity. Here we extend our previous work and propose a nonparametric Bayesian approach to infer rat hippocampal population codes during spatial navigation. To tackle the model selection problem, we leverage a nonparametric Bayesian model. Specifically, to analyze rat hippocampal ensemble spiking activity, we apply a hierarchical Dirichlet process-hidden Markov model (HDP-HMM) using two Bayesian inference methods, one based on Markov chain Monte Carlo (MCMC) and the other based on variational Bayes (VB). We demonstrate the effectiveness of our Bayesian approaches on recordings from a freely-behaving rat navigating in an…

Looking into the past: Eye-tracking mental simulation in physical inference

Ari Beller, Yingchen Xu, Scott W. Linderman +1 more

Computer Vision and Pattern Recognition · eScholarship (California Digital Library) · · cited 5× · DOI ↗ · more like this ·

Mental simulation is a powerful cognitive capacity that underlies people's ability to draw inferences about what happened in the past from the present. Recent work suggests that eyetracking can be used as a window through which one can study the process of mental simulation in intuitive physics tasks. In our experiment, participants have to figure out in which of three holes a ball was dropped in a virtual Plinko box. We develop a computational model of human intuitive physical reasoning in Plinko that runs repeated simulations in a noisy physics simulator in order to infer in which hole the ball was dropped. We evaluate our model's behavior against multiple human data signals: trial judgments, response times, and eye-movement data. We find that a model that sequentially samples simulations while balancing uncertainty and reward best explains the patterns of participant behavior we…

Cross-corpora unsupervised learning of trajectories in autism spectrum disorders

Huseyin Melih Elibol, Vincent Nguyen, Scott W. Linderman +3 more

Artificial Intelligence · Journal of Machine Learning Research · · cited 5× · OpenAlex ↗ · more like this ·

Patients with developmental disorders, such as autism spectrum disorder (ASD), present with symptoms that change with time even if the named diagnosis remains fixed. For example, language impairments may present as delayed speech in a toddler and difficulty reading in a school-age child. Characterizing these trajectories is important for early treatment. However, deriving these trajectories from observational sources is challenging: electronic health records only reflect observations of patients at irregular intervals and only record what factors are clinically relevant at the time of observation. Meanwhile, caretakers discuss daily developments and concerns on social media. In this work, we present a fully unsupervised approach for learning disease trajectories from incomplete medical records and social media posts, including cases in which we have only a single observation of each…

Modeling Latent Neural Dynamics with Gaussian Process Switching Linear Dynamical Systems

Amber Hu, David Zoltowski, Aditya Nair +3 more

q-bio.NC · · cited 3× · arXiv ↗ · more like this ·

Understanding how the collective activity of neural populations relates to computation and ultimately behavior is a key goal in neuroscience. To this end, statistical methods which describe high-dimensional neural time series in terms of low-dimensional latent dynamics have played a fundamental role in characterizing neural systems. Yet, what constitutes a successful method involves two opposing criteria: (1) methods should be expressive enough to capture complex nonlinear dynamics, and (2) they should maintain a notion of interpretability often only warranted by simpler linear models. In this paper, we develop an approach that balances these two objectives: the Gaussian Process Switching Linear Dynamical System (gpSLDS). Our method builds on previous work modeling the latent state evolution via a stochastic differential equation whose nonlinear dynamics are described by a Gaussian…

Language models suffer from a curse of ambiguity

Nicolas Zucchet, Hyun Dong Lee, Scott Linderman

cs.CL · · cited 2× · arXiv ↗ · more like this ·

Large language models increasingly rely on sampling as a driver of their own improvement, making the fidelity of their learned distributions more critical than ever. Yet, not all distributions are equally easy to learn. In this work, we identify a curse of ambiguity: in large language models, and more broadly in all neural networks that produce discrete probability distributions, the more ambiguous a next-token distribution is, the harder it is to learn accurately. Through an extensive theoretical analysis, we trace this curse to architectural and learning roots. More ambiguous distributions require more capacity to be stored, larger embeddings to be represented, more steps to be fitted, and amplify token-sampling noise. We validate these findings on synthetic tasks with controlled ground truth and observe the same signatures in language models trained on real data. Our results provide…

Dose-response modeling in high-throughput cancer drug screenings: An end-to-end approach

Wesley Tansey, Kathy Li, Haoran Zhang +4 more

stat.AP · · cited 2× · arXiv ↗ · more like this ·

Personalized cancer treatments based on the molecular profile of a patient's tumor are an emerging and exciting class of treatments in oncology. As genomic tumor profiling is becoming more common, targeted treatments to specific molecular alterations are gaining traction. To discover new potential therapeutics that may apply to broad classes of tumors matching some molecular pattern, experimentalists and pharmacologists rely on high-throughput, in-vitro screens of many compounds against many different cell lines. We propose a hierarchical Bayesian model of how cancer cell lines respond to drugs in these experiments and develop a method for fitting the model to real-world high-throughput screening data. Through a case study, the model is shown to capture nontrivial associations between molecular features and drug response, such as requiring both wild type TP53 and overexpression of MDM2…

Learning Structured Neural Dynamics From Single Trial Population Recording

Josue Nassar, Scott W. Linderman, Yuan Zhao +2 more

Artificial Intelligence · 2018 52nd Asilomar Conference on Signals, Systems, and Computers · · cited 2× · DOI ↗ · more like this ·

To understand the complex nonlinear dynamics of neural circuits, we fit a structured state-space model called tree-structured recurrent switching linear dynamical system (TrSLDS) to noisy high-dimensional neural time series. TrSLDS is a multi-scale hierarchical generative model for the state-space dynamics where each node of the latent tree captures locally linear dynamics. TrSLDS can be learned efficiently and in a fully Bayesian manner using Gibbs sampling. We showcase TrSLDS’ potential of inferring low-dimensional interpretable dynamical systems on a variety of examples.

Towards Scalable and Stable Parallelization of Nonlinear RNNs

Xavier Gonzalez, Andrew Warrington, Jimmy T. H. Smith +1 more

cs.LG · · cited 1× · arXiv ↗ · more like this ·

Transformers and linear state space models can be evaluated in parallel on modern hardware, but evaluating nonlinear RNNs appears to be an inherently sequential problem. Recently, however, Lim et al. '24 developed an approach called DEER, which evaluates nonlinear RNNs in parallel by posing the states as the solution to a fixed-point problem. They derived a parallel form of Newton's method to solve the fixed-point problem and achieved significant speedups over sequential evaluation. However, the computational complexity of DEER is cubic in the state size, and the algorithm can suffer from numerical instability. We address these limitations with two novel contributions. To reduce the computational complexity, we apply quasi-Newton approximations and show they converge comparably to Newton, use less memory, and are faster. To stabilize DEER, we leverage a connection between the…