arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Kaiser

5,861 papers · 1877–2026

1967: 21968: 51969: 71970: 41971: 91972: 41973: 51974: 61975: 41976: 41977: 81978: 131979: 91980: 81981: 81982: 61983: 61984: 111985: 131986: 131987: 191988: 321989: 211990: 181991: 391992: 291993: 381994: 451995: 441996: 591997: 531998: 471999: 642000: 712001: 882002: 1272003: 1222004: 1162005: 1302006: 1092007: 1382008: 1452009: 1432010: 1672011: 1802012: 1682013: 1942014: 2512015: 2372016: 2382017: 2102018: 2392019: 2212020: 2842021: 2692022: 2462023: 2802024: 3132025: 3392026: 134
1967peak 339 in 20252026

Frequent co-authors

Papers

showing 1–25 of 5,861 RSS

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar +5 more

cs.CL · · cited 194,990× · arXiv ↗ · more like this ·

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a…

GPT-4 Technical Report

OpenAI, Josh Achiam, Steven Adler +278 more

cs.CL · · cited 27,487× · arXiv ↗ · more like this ·

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.

An Index of Factorial Simplicity

Henry F. Kaiser

Computational Theory and Mathematics · Psychometrika · · cited 14,243× · DOI ↗ · more like this ·

An index of factorial simplicity, employing the quartimax transformational criteria of Carroll, Wrigley and Neuhaus, and Saunders, is developed. This index is both for each row separately and for a factor pattern matrix as a whole. The index varies between zero and one. The problem of calibrating the index is discussed.

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Martín Abadi, Ashish Agarwal, Paul Barham +37 more

cs.DC · · cited 11,866× · arXiv ↗ · more like this ·

TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction…

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun +55 more

cs.LG · · cited 11,833× · arXiv ↗ · more like this ·

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code…

Training Verifiers to Solve Math Word Problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian +9 more

cs.LG · · cited 11,014× · arXiv ↗ · more like this ·

State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Yonghui Wu, Mike Schuster, Zhifeng Chen +28 more

cs.CL · · cited 7,404× · arXiv ↗ · more like this ·

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To…

Code Saturation Versus Meaning Saturation

Monique Hennink, Bonnie N. Kaiser, Vincent C. Marconi

Sociology and Political Science · Qualitative Health Research · · cited 4,187× · DOI ↗ · more like this ·

Saturation is a core guiding principle to determine sample sizes in qualitative research, yet little methodological research exists on parameters that influence saturation. Our study compared two approaches to assessing saturation: code saturation and meaning saturation. We examined sample sizes needed to reach saturation in each approach, what saturation meant, and how to assess saturation. Examining 25 in-depth interviews, we found that code saturation was reached at nine interviews, whereby the range of thematic issues was identified. However, 16 to 24 interviews were needed to reach meaning saturation where we developed a richly textured understanding of issues. Thus, code saturation may indicate when researchers have "heard it all," but meaning saturation is needed to "understand it all." We used our results to develop parameters that influence saturation, which may be used to…

Wireless integrated network sensors

Gregory J. Pottie, William J. Kaiser

Computer Networks and Communications · Communications of the ACM · · cited 3,227× · DOI ↗ · more like this ·

For pervasive computing performance, exploit the physical limits of these densely distributed networks of embedded sensors, controls, and processors. W ireless integrated network sensors (WINS) provide distributed network and Internet access to sensors, controls, and processors deeply embedded in equipment, facilities, and the environment. The WINS network represents a new monitoring and control capability for applications in such industries as transportation, manufacturing, health care, environmental oversight, and safety and security. WINS combine microsensor technology and low-power signal processing, computation, and low-cost wireless networking in a compact system.

Reformer: The Efficient Transformer

Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya

cs.LG · · cited 3,117× · arXiv ↗ · more like this ·

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

Clustering in real space and in redshift space

Nick Kaiser

Artificial Intelligence · Monthly Notices of the Royal Astronomical Society · · cited 2,744× · DOI ↗ · more like this ·

Peculiar velocities distort the clustering pattern in red shift space on all scales. Four consequences of this are: (i) The acceleration vector derived by summing the inverse squared redshifts of galaxies differs significantly from the true acceleration even in linear theory. Estimates of Ω obtained in this manner are only reliable for small Ω. (ii) The power spectrum of large-scale clustering has a quadrupole anisotropy, providing a way to estimate Ω. We calculated, for various assumed power spectra, the line-of-sight correlation function in redshift space, <it>ξ</it><inf><it>υ</it>.</inf> We find that <it>ξ</it><inf><it>υ</it></inf> may display a strong anticorrelation feature that has no counterpart in real space. (iii) The density contrast of the local supercluster will…

Rethinking Attention with Performers

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan +10 more

cs.LG · · cited 2,501× · arXiv ↗ · more like this ·

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong…

Image Transformer

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit +4 more

cs.CV · · cited 1,955× · arXiv ↗ · more like this ·

Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also…

Regularizing Neural Networks by Penalizing Confident Output Distributions

Gabriel Pereyra, George Tucker, Jan Chorowski +2 more

cs.NE · · cited 1,293× · arXiv ↗ · more like this ·

We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.

On a simple algorithm to calculate the 'energy' of a signal

J. Kaiser

Artificial Intelligence · International Conference on Acoustics, Speech, and Signal Processing · · cited 1,143× · DOI ↗ · more like this ·

A simple algorithm is derived that permits on-the-fly calculation of the energy required to generate, in a certain sense, a signal. The results of applying this algorithm to a number of well-known signals are shown. Some of the invariance and noise properties of the algorithm are derived and verified by simulation. The implementation of the algorithm and its application to speech processing are briefly discussed.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

What Influences Saturation? Estimating Sample Sizes in Focus Group Research

Monique Hennink, Bonnie N. Kaiser, Mary Beth Weber

Sociology and Political Science · Qualitative Health Research · · cited 1,134× · DOI ↗ · more like this ·

Saturation is commonly used to determine sample sizes in qualitative research, yet there is little guidance on what influences saturation. We aimed to assess saturation and identify parameters to estimate sample sizes for focus group studies in advance of data collection. We used two approaches to assess saturation in data from 10 focus group discussions. Four focus groups were sufficient to identify a range of new issues (code saturation), but more groups were needed to fully understand these issues (meaning saturation). Group stratification influenced meaning saturation, whereby one focus group per stratum was needed to identify issues; two groups per stratum provided a more comprehensive understanding of issues, but more groups per stratum provided little additional benefit. We identify six parameters influencing saturation in focus group data: study purpose, type of codes, group…

Universal Transformers

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals +2 more

cs.CL · · cited 1,089× · arXiv ↗ · more like this ·

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a…

Model-Based Reinforcement Learning for Atari

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos +11 more

cs.LG · · cited 1,017× · arXiv ↗ · more like this ·

Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -- substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a…

Grammar as a Foreign Language

Oriol Vinyals, Lukasz Kaiser, Terry Koo +3 more

cs.CL · · cited 944× · arXiv ↗ · more like this ·

Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.

Generating Wikipedia by Summarizing Long Sequences

Peter J. Liu, Mohammad Saleh, Etienne Pot +4 more

cs.CL · · cited 881× · arXiv ↗ · more like this ·

We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.

Multi-task Sequence to Sequence Learning

Minh-Thang Luong, Quoc V. Le, Ilya Sutskever +2 more

cs.LG · · cited 841× · arXiv ↗ · more like this ·

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over…

Leadership, followership, and evolution: Some lessons from the past.

Mark van Vugt, Robert Hogan, Robert B. Kaiser

Sociology and Political Science · American Psychologist · · cited 779× · DOI ↗ · more like this ·

This article analyzes the topic of leadership from an evolutionary perspective and proposes three conclusions that are not part of mainstream theory. First, leading and following are strategies that evolved for solving social coordination problems in ancestral environments, including in particular the problems of group movement, intragroup peacekeeping, and intergroup competition. Second, the relationship between leaders and followers is inherently ambivalent because of the potential for exploitation of followers by leaders. Third, modern organizational structures are sometimes inconsistent with aspects of our evolved leadership psychology, which might explain the alienation and frustration of many citizens and employees. The authors draw several implications of this evolutionary analysis for leadership theory, research, and practice.

Energy separation in signal modulations with application to speech analysis

Petros Maragos, J. Kaiser, Thomas F. Quatieri

Signal Processing · IEEE Transactions on Signal Processing · · cited 766× · DOI ↗ · more like this ·

An efficient solution to the fundamental problem of estimating the time-varying amplitude envelope and instantaneous frequency of a real-valued signal that has both an AM and FM structure is provided. Nonlinear combinations of instantaneous signal outputs from the energy operator are used to separate its output energy product into its AM and FM components. The theoretical analysis is done first for continuous-time signals. Then several efficient algorithms are developed and compared for estimating the amplitude envelope and instantaneous frequency of discrete-time AM-FM signals. These energy separation algorithms are used to search for modulations in speech resonances, which are modeled using AM-FM signals to account for time-varying amplitude envelopes and instantaneous frequencies. The experimental results provide evidence that bandpass-filtered speech signals around speech formants…

Protecting Respondent Confidentiality in Qualitative Research

Karen Kaiser

Sociology and Political Science · Qualitative Health Research · · cited 758× · DOI ↗ · more like this ·

For qualitative researchers, maintaining respondent confidentiality while presenting rich, detailed accounts of social life presents unique challenges. These challenges are not adequately addressed in the literature on research ethics and research methods. Using an example from a study of breast cancer survivors, I argue that by carefully considering the audience for one's research and by reenvisioning the informed consent process, qualitative researchers can avoid confidentiality dilemmas that might otherwise lead them not to report rich, detailed data.