arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

⁄ Gordon

7,995 papers · 1805–2026

1967: 161968: 141969: 211970: 231971: 161972: 261973: 261974: 301975: 241976: 281977: 281978: 351979: 381980: 351981: 371982: 371983: 441984: 301985: 351986: 511987: 591988: 421989: 621990: 561991: 531992: 661993: 871994: 921995: 1111996: 1091997: 1141998: 1151999: 1312000: 1272001: 1492002: 1872003: 2172004: 2222005: 2612006: 2732007: 2402008: 2472009: 2632010: 2732011: 2832012: 3942013: 3672014: 3142015: 2552016: 1942017: 1692018: 2242019: 2402020: 2302021: 2452022: 2552023: 2482024: 2602025: 222026: 2
1967peak 394 in 20122026

Frequent co-authors

Papers

showing 1–25 of 7,995 RSS

GPT-4 Technical Report

OpenAI, Josh Achiam, Steven Adler +278 more

cs.CL · · cited 27,487× · arXiv ↗ · more like this ·

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.

LAION-5B: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont, Richard Vencu +13 more

cs.CV · · cited 5,591× · arXiv ↗ · more like this ·

Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B -…

In-Datacenter Performance Analysis of a Tensor Processing Unit

Norman P. Jouppi, Cliff Young, Nishant Patil +72 more

cs.AR · · cited 5,472× · arXiv ↗ · more like this ·

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU…

EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces

Vernon J. Lawhern, Amelia J. Solon, Nicholas R. Waytowich +3 more

cs.LG · · cited 4,904× · arXiv ↗ · more like this ·

Brain computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct characteristics of its expected EEG control signal, limiting its application to that specific signal. Convolutional Neural Networks (CNNs), which have been used in computer vision and speech recognition, have successfully been applied to EEG-based BCIs; however, they have mainly been applied to single BCI paradigms and thus it remains unclear how these architectures generalize to other paradigms. Here, we ask if we can design a single CNN architecture to accurately classify EEG signals from different BCI paradigms, while simultaneously being as compact as possible. In this…

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell

cs.LG · · cited 4,696× · arXiv ↗ · more like this ·

Sequential prediction problems such as imitation learning, where future observations depend on previous predictions (actions), violate the common i.i.d. assumptions made in statistical learning. This leads to poor performance in theory and often in practice. Some recent approaches provide stronger guarantees in this setting, but remain somewhat unsatisfactory as they train either non-stationary or stochastic policies and require a large number of iterations. In this paper, we propose a new iterative algorithm, which trains a stationary deterministic policy, that can be seen as a no regret algorithm in an online learning setting. We show that any such no regret algorithm, combined with additional reduction assumptions, must find a policy with good performance under the distribution of observations it induces in such sequential settings. We demonstrate that this new approach outperforms…

Implicit Neural Representations with Periodic Activation Functions

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman +2 more

cs.CV · · cited 3,987× · arXiv ↗ · more like this ·

Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and…

The science of fake news

David M. J. Lazer, Matthew A. Baum, Yochai Benkler +13 more

cs.CY · · cited 3,922× · arXiv ↗ · more like this ·

Fake news emerged as an apparent global problem during the 2016 U.S. Presidential election. Addressing it requires a multidisciplinary effort to define the nature and extent of the problem, detect fake news in real time, and mitigate its potentially harmful effects. This will require a better understanding of how the Internet spreads content, how people process news, and how the two interact. We review the state of knowledge in these areas and discuss two broad potential mitigation strategies: better enabling individuals to identify fake news, and intervention within the platforms to reduce the attention given to fake news. The cooperation of Internet platforms (especially Facebook, Google, and Twitter) with researchers will be critical to understanding the scale of the issue and the effectiveness of possible interventions.

Gemma 2: Improving Open Language Models at a Practical Size

Gemma Team, Morgane Riviere, Shreya Pathak +195 more

cs.CL · · cited 2,513× · arXiv ↗ · more like this ·

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Averaging Weights Leads to Wider Optima and Better Generalization

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov +2 more

cs.LG · · cited 2,186× · arXiv ↗ · more like this ·

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

Discrete sequential boundaries for clinical trials

K. K. Gordon Lan, David L. DeMets

Statistics and Probability · Biometrika · · cited 1,740× · DOI ↗ · more like this ·

Pocock (1977), O'Brien & Fleming (1979) and Slud & Wei (1982) have proposed different methods to construct discrete sequential boundaries for clinical trials. These methods require that the total number of decision times be specified in advance. In the present paper, we propose a more flexible way to construct discrete sequential boundaries. The method is based on the choice of a function, α*(t), which characterizes the rate at which the error level α is spent. The boundary at a decision time is determined by α*(t), and by past and current decision times, but does not depend on the future decision times or the total number of decision times.

Social Capital, Structural Holes and the Formation of an Industry Network

Gordon Walker, Bruce Kogut, Weijian Shan

Sociology and Political Science · Organization Science · · cited 1,701× · DOI ↗ · more like this ·

The formation of a network is determined by the opposition of two forces. The first is the reproduction of network structure as a general social resource for network members. The second is the alteration of network structure by entrepreneurs for their own benefit. The idea of reproduction is a conventional one in organizational sociology but has taken on increased importance due to the work of Bourdieu and Coleman. In contrast, Burt stresses the entrepreneurship of individual agents in exploiting structural holes that lie between constrained positions. Though complementary, the theories of social capital and structural holes have fundamentally different implications for network formation. This paper investigates these theories by examining empirically the formation of the interorganizational network among biotechnology firms. We propose that network structure determines the frequency…

Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman +6 more

cs.LG · · cited 1,637× · arXiv ↗ · more like this ·

Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution…

AI2-THOR: An Interactive 3D Environment for Visual AI

Eric Kolve, Roozbeh Mottaghi, Winson Han +10 more

cs.CV · · cited 1,572× · arXiv ↗ · more like this ·

We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to facilitate building visually intelligent models and push the research forward in this domain.

Efficient Geometry-aware 3D Generative Adversarial Networks

Eric R. Chan, Connor Z. Lin, Matthew A. Chan +9 more

cs.CV · · cited 1,555× · arXiv ↗ · more like this ·

Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute-intensive or make approximations that are not 3D-consistent; the former limits quality and resolution of the generated images and the latter adversely affects multi-view consistency and shape quality. In this work, we improve the computational efficiency and image quality of 3D GANs without overly relying on these approximations. We introduce an expressive hybrid explicit-implicit network architecture that, together with other design choices, synthesizes not only high-resolution multi-view-consistent images in real time but also produces high-quality 3D geometry. By decoupling feature generation and neural rendering, our framework is able to leverage state-of-the-art 2D CNN…

GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

Jacob R. Gardner, Geoff Pleiss, David Bindel +2 more

cs.LG · · cited 1,527× · arXiv ↗ · more like this ·

Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from $O(n^3)$ to $O(n^2)$. Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and…

Simulated Quantum Computation of Molecular Energies

Alán Aspuru-Guzik, Anthony D. Dutoi, Peter J. Love +1 more

quant-ph · · cited 1,525× · arXiv ↗ · more like this ·

The calculation time for the energy of atoms and molecules scales exponentially with system size on a classical computer but polynomially using quantum algorithms. We demonstrate that such algorithms can be applied to problems of chemical interest using modest numbers of quantum bits. Calculations of the water and lithium hydride molecular ground-state energies have been carried out on a quantum computer simulator using a recursive phase-estimation algorithm. The recursive algorithm reduces the number of quantum bits required for the readout register from about 20 to 4. Mappings of the molecular wave function to the quantum bits are described. An adiabatic method for the preparation of a good approximate ground-state wave function is described and demonstrated for a stretched hydrogen molecule. The number of quantum bits required scales linearly with the number of basis functions, and…

Human Motion Diffusion Model

Guy Tevet, Sigal Raab, Brian Gordon +3 more

cs.CV · · cited 1,443× · arXiv ↗ · more like this ·

Natural and expressive human motion generation is the holy grail of computer animation. It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in expressiveness. Diffusion models, which have already shown remarkable generative capabilities in other domains, are promising candidates for human motion due to their many-to-many nature, but they tend to be resource hungry and hard to control. In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for the human motion domain. MDM is transformer-based, combining insights from motion generation literature. A notable design-choice is the prediction of the sample, rather than the noise, in each diffusion…

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein

cs.CV · · cited 1,443× · arXiv ↗ · more like this ·

Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation…

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

Mohit Shridhar, Jesse Thomason, Daniel Gordon +5 more

cs.CV · · cited 1,217× · arXiv ↗ · more like this ·

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level language instructions like "Walk to the coffee maker on the right." ALFRED tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly…

Accelerating 3D Deep Learning with PyTorch3D

Nikhila Ravi, Jeremy Reizenstein, David Novotny +4 more

cs.CV · · cited 1,199× · arXiv ↗ · more like this ·

Deep learning has significantly improved 2D image recognition. Extending into 3D may advance many new applications including autonomous vehicles, virtual and augmented reality, authoring 3D content, and even improving 2D recognition. However despite growing interest, 3D deep learning remains relatively underexplored. We believe that some of this disparity is due to the engineering challenges involved in 3D deep learning, such as efficiently processing heterogeneous data and reframing graphics operations to be differentiable. We address these challenges by introducing PyTorch3D, a library of modular, efficient, and differentiable operators for 3D deep learning. It includes a fast, modular differentiable renderer for meshes and point clouds, enabling analysis-by-synthesis approaches. Compared with other differentiable renderers, PyTorch3D is more modular and efficient, allowing users to…

Deep Kernel Learning

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov +1 more

cs.LG · · cited 1,122× · arXiv ↗ · more like this ·

We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods. Specifically, we transform the inputs of a spectral mixture base kernel with a deep architecture, using local kernel interpolation, inducing points, and structure exploiting (Kronecker and Toeplitz) algebra for a scalable kernel representation. These closed-form kernels can be used as drop-in replacements for standard kernels, with benefits in expressive power and scalability. We jointly learn the properties of these kernels through the marginal likelihood of a Gaussian process. Inference and learning cost $O(n)$ for $n$ training points, and predictions cost $O(1)$ per test point. On a large and diverse collection of applications, including a dataset with 2 million examples, we show improved performance over scalable Gaussian…

Chronos: Learning the Language of Time Series

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen +15 more

cs.LG · · cited 1,109× · arXiv ↗ · more like this ·

We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new…

Perspective-taking: Decreasing stereotype expression, stereotype accessibility, and in-group favoritism.

Adam D. Galinsky, Gordon B. Moskowitz

Sociology and Political Science · Journal of Personality and Social Psychology · · cited 1,108× · DOI ↗ · more like this ·

Using 3 experiments, the authors explored the role of perspective-taking in debiasing social thought. In the 1st 2 experiments, perspective-taking was contrasted with stereotype suppression as a possible strategy for achieving stereotype control. In Experiment 1, perspective-taking decreased stereotypic biases on both a conscious and a nonconscious task. In Experiment 2, perspective-taking led to both decreased stereotyping and increased overlap between representations of the self and representations of the elderly, suggesting activation and application of the self-concept in judgments of the elderly. In Experiment 3, perspective-taking reduced evidence of in-group bias in the minimal group paradigm by increasing evaluations of the out-group. The role of self-other overlap in producing prosocial outcomes and the separation of the conscious, explicit effects from the nonconscious…

KEA: Practical Automatic Keyphrase Extraction

Ian H. Witten, Gordon W. Paynter, Eibe Frank +2 more

cs.DL · · cited 1,086× · arXiv ↗ · more like this ·

Keyphrases provide semantic metadata that summarize and characterize documents. This paper describes Kea, an algorithm for automatically extracting keyphrases from text. Kea identifies candidate keyphrases using lexical methods, calculates feature values for each candidate, and uses a machine-learning algorithm to predict which candidates are good keyphrases. The machine learning scheme first builds a prediction model using training documents with known keyphrases, and then uses the model to find keyphrases in new documents. We use a large test corpus to evaluate Kea's effectiveness in terms of how many author-assigned keyphrases are correctly identified. The system is simple, robust, and publicly available.

Politicization of Science in the Public Sphere

Gordon Gauchat

Sociology and Political Science · American Sociological Review · · cited 1,077× · DOI ↗ · more like this ·

This study explores time trends in public trust in science in the United States from 1974 to 2010. More precisely, I test Mooney’s (2005) claim that conservatives in the United States have become increasingly distrustful of science. Using data from the 1974 to 2010 General Social Survey, I examine group differences in trust in science and group-specific change in these attitudes over time. Results show that group differences in trust in science are largely stable over the period, except for respondents identifying as conservative. Conservatives began the period with the highest trust in science, relative to liberals and moderates, and ended the period with the lowest. The patterns for science are also unique when compared to public trust in other secular institutions. Results show enduring differences in trust in science by social class, ethnicity, gender, church attendance, and region.…