Author

Carl

7,992 papers · 1827–2026

1967peak 350 in 20252026

Frequent co-authors

Papers

showing 1–25 of 7,992 RSS

GPT-4 Technical Report

OpenAI, Josh Achiam, Steven Adler +278 more

cs.CL · March 2023 · cited 27,487× · arXiv ↗ · more like this ·

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.

The Llama 3 Herd of Models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +558 more

cs.AI · July 2024 · cited 19,265× · arXiv ↗ · more like this ·

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs…

Bootstrap your own latent: A new approach to self-supervised Learning

Jean-Bastien Grill, Florian Strub, Florent Altché +11 more

cs.LG · June 2020 · cited 9,145× · arXiv ↗ · more like this ·

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation…

The Anatomy of the Grid - Enabling Scalable Virtual Organizations

Ian Foster, Carl Kesselman, Steven Tuecke

cs.AR · 29 Mar 2001 · cited 8,203× · arXiv ↗ · more like this ·

"Grid" computing has emerged as an important new field, distinguished from conventional distributed computing by its focus on large-scale resource sharing, innovative applications, and, in some cases, high-performance orientation. In this article, we define this new field. First, we review the "Grid problem," which we define as flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources-what we refer to as virtual organizations. In such settings, we encounter unique authentication, authorization, resource access, resource discovery, and other challenges. It is this class of problem that is addressed by Grid technologies. Next, we present an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling…

Empirical Analysis of Predictive Algorithms for Collaborative Filtering

John S. Breese, David Heckerman, Carl Kadie

cs.IR · 30 Jan 2013 · cited 5,986× · arXiv ↗ · more like this ·

Collaborative filtering or recommender systems use a database about user preferences to predict additional topics or products a new user might like. In this paper we describe several algorithms designed for this task, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods. We compare the predictive accuracy of the various methods in a set of representative problem domains. We use two basic classes of evaluation metrics. The first characterizes accuracy over a set of individual predictions in terms of average absolute deviation. The second estimates the utility of a ranked list of suggested items. This metric uses an estimate of the probability that a user will see a recommendation in an ordered list. Experiments were run for datasets associated with 3 application areas, 4 experimental protocols, and the 2 evaluation…

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team, Petko Georgiev, Ving Ian Lei +1134 more

cs.CL · March 2024 · cited 4,033× · arXiv ↗ · more like this ·

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's…

The Approximation of One Matrix by Another of Lower Rank

Carl Eckart, Gale Young

Computational Theory and Mathematics · Psychometrika · September 1936 · cited 3,795× · DOI ↗ · more like this ·

The mathematical problem of approximating one matrix by another of lower rank is closely related to the fundamental postulate of factor-theory. When formulated as a least-squares problem, the normal equations cannot be immediately written down, since the elements of the approximate matrix are not independent of one another. The solution of the problem is simplified by first expressing the matrices in a canonic form. It is found that the problem always has a solution which is usually unique. Several conclusions can be drawn from the form of this solution. A hypothetical interpretation of the canonic components of a score matrix is discussed.

Matrix analysis and applied linear algebra

Meyer, Carl D.

Computational Theory and Mathematics · Choice Reviews Online · May 2001 · cited 3,411× · DOI ↗ · more like this ·

Preface 1. Linear equations 2. Rectangular systems and echelon forms 3. Matrix algebra 4. Vector spaces 5. Norms, inner products, and orthogonality 6. Determinants 7. Eigenvalues and Eigenvectors 8. Perron-Frobenius theory of nonnegative matrices Index.

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Dario Amodei, Rishita Anubhai, Eric Battenberg +31 more

cs.CL · 8 Dec 2015 · cited 3,201× · arXiv ↗ · more like this ·

We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, resulting in a 7x speedup over our previous system. Because of this efficiency, experiments that previously took weeks now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can…

The Physiology of the Grid An Open Grid Services Architecture for Distributed Systems Integration

Ian Foster, Carl Kesselman, Jeffrey M. Nick +1 more

Computer Networks and Communications · 2002 · cited 3,129× · OpenAlex ↗ · more like this ·

In both e-business and e-science, we often need to integrate services across distributed, heterogeneous, dynamic &quot;virtual organizations&quot; formed from the disparate resources within a single enterprise and/or from external resource sharing and service provider relationships. This integration can be technically challenging because of the need to achieve various qualities of service when running on top of different native platforms. We present an Open Grid Services Architecture that addresses these challenges. Building on concepts and technologies from the Grid and Web services communities, this architecture defines a uniform exposed service semantics (the Grid service); defines standard mechanisms for creating, naming, and discovering transient Grid service instances; provides location transparency and multiple protocol bindings for service instances; and supports…

Unsupervised Visual Representation Learning by Context Prediction

Carl Doersch, Abhinav Gupta, Alexei A. Efros

cs.CV · May 2015 · cited 3,010× · arXiv ↗ · more like this ·

This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework and provides a significant boost over a randomly-initialized…

Qualitative data: an introduction to coding and analysis

Auerbach, Carl F., Silverstein, Louise B.

Education · Choice Reviews Online · March 2004 · cited 2,671× · DOI ↗ · more like this ·

Preface AcknowledgmentsPart I: Getting into Qualitative Research1. Introducing Qualitative Hypothesis-Generating Research: The Yeshiva University Fatherhood ProjectPart II: Planning Your First Research Study2. Designing Hypothesis-Generating Research: The Haitian Fathers Study3. Qualitative and Quantitative Research as Complementary Strategies Part III: Analyzing Your First Research Study4. Coding 1: The Basic Ideas 5. Coding 2: The Mechanics, Phase 1: Making the Text Manageable 6. Coding 2: The Mechanics, Phase 2: Hearing What Was Said 7. Coding 2: The Mechanics, Phase 3: Developing Theory 8. Convincing Other People: The Issues Formerly Known as Reliability, Validity, and Generalizability Part IV: Designing and Analyzing Your Next Research Study9. Designing Your Next Study Using Theoretical Sampling: The Promise Keeper Fathers 10. Analyzing Your Next Study Using Elaborative Coding: The…

Gemma 2: Improving Open Language Models at a Practical Size

Gemma Team, Morgane Riviere, Shreya Pathak +195 more

cs.CL · August 2024 · cited 2,513× · arXiv ↗ · more like this ·

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Deep Speech: Scaling up end-to-end speech recognition

Awni Hannun, Carl Case, Jared Casper +8 more

cs.CL · 19 Dec 2014 · cited 2,286× · arXiv ↗ · more like this ·

We present a state-of-the-art speech recognition system developed using end-to-end deep learning. Our architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, our system does not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learns a function that is robust to such effects. We do not need a phoneme dictionary, nor even the concept of a "phoneme." Key to our approach is a well-optimized RNN training system that uses multiple GPUs, as well as a set of novel data synthesis techniques that allow us to efficiently obtain a large amount of varied data for training. Our system, called Deep Speech, outperforms previously published results on the widely…

Tutorial on Variational Autoencoders

Carl Doersch

stat.ML · June 2016 · cited 2,052× · arXiv ↗ · more like this ·

In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. VAEs are appealing because they are built on top of standard function approximators (neural networks), and can be trained with stochastic gradient descent. VAEs have already shown promise in generating many kinds of complicated data, including handwritten digits, faces, house numbers, CIFAR images, physical models of scenes, segmentation, and predicting the future from static images. This tutorial introduces the intuitions behind VAEs, explains the mathematics behind them, and describes some empirical behavior. No prior knowledge of variational Bayesian methods is assumed.

Grid services for distributed system integration

Ian Foster, Carl Kesselman, J.M. Nick +1 more

Computer Networks and Communications · Computer · June 2002 · cited 1,959× · DOI ↗ · more like this ·

Increasingly, computing addresses collaboration, data sharing, and interaction modes that involve distributed resources, resulting in an increased focus on the interconnection of systems both within and across enterprises. These evolutionary pressures have led to the development of Grid technologies. The authors' work focuses on the nature of the services that respond to protocol messages. Grid provides an extensible set of services that can be aggregated in various ways to meet the needs of virtual organizations, which themselves can be defined in part by the services they operate and share.

Zero-1-to-3: Zero-shot One Image to 3D Object

Ruoshi Liu, Rundi Wu, Basile Van Hoorick +3 more

cs.CV · 20 Mar 2023 · cited 1,878× · arXiv ↗ · more like this ·

We introduce Zero-1-to-3, a framework for changing the camera viewpoint of an object given just a single RGB image. To perform novel view synthesis in this under-constrained setting, we capitalize on the geometric priors that large-scale diffusion models learn about natural images. Our conditional diffusion model uses a synthetic dataset to learn controls of the relative camera viewpoint, which allow new images to be generated of the same object under a specified camera transformation. Even though it is trained on a synthetic dataset, our model retains a strong zero-shot generalization ability to out-of-distribution datasets as well as in-the-wild images, including impressionist paintings. Our viewpoint-conditioned diffusion approach can further be used for the task of 3D reconstruction from a single image. Qualitative and quantitative experiments show that our method significantly…

A Unifying View of Sparse Approximate Gaussian Process Regression

Joaquin Quiñonero-Candela, Carl Edward Rasmussen

Artificial Intelligence · MPG.PuRe (Max Planck Society) · December 2005 · cited 1,784× · DOI ↗ · more like this ·

We provide a new unifying view, including all existing proper probabilistic\nsparse approximations for Gaussian process regression. Our approach relies on\nexpressing the effective prior which the methods are using. This\nallows new insights to be gained, and highlights the relationship between\nexisting methods. It also allows for a clear theoretically justified ranking\nof the closeness of the known approximations to the corresponding full GPs.\nFinally we point directly to designs of new better sparse approximations,\ncombining the best of the existing strategies, within attractive\ncomputational constraints.

Generating Videos with Scene Dynamics

Carl Vondrick, Hamed Pirsiavash, Antonio Torralba

cs.CV · September 2016 · cited 1,596× · arXiv ↗ · more like this ·

We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.

Data-Efficient Image Recognition with Contrastive Predictive Coding

Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw +4 more

cs.CV · May 2019 · cited 1,584× · arXiv ↗ · more like this ·

Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained…

VideoBERT: A Joint Model for Video and Language Representation Learning

Chen Sun, Austin Myers, Carl Vondrick +2 more

cs.CV · April 2019 · cited 1,442× · arXiv ↗ · more like this ·

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we…

Parametric Empirical Bayes Inference: Theory and Applications

Carl N. Morris

Statistics and Probability · Journal of the American Statistical Association · March 1983 · cited 1,393× · DOI ↗ · more like this ·

Abstract This article reviews the state of multiparameter shrinkage estimators with emphasis on the empirical Bayes viewpoint, particularly in the case of parametric prior distributions. Some successful applications of major importance are considered. Recent results concerning estimates of error and confidence intervals are described and illustrated with data.

Education and mind in the knowledge age

Bereiter, Carl

Education · Choice Reviews Online · December 2002 · cited 1,275× · DOI ↗ · more like this ·

Online Publication Date: 01 December 2003To cite this Article: (2003) 'Resource reviews', Teaching Education, 14:3, 333 - 343To link to this article: DOI: 10.1080/1047621032000135221URL: http://dx.doi.org/10.1080/1047621032000135221PLEASE SCROLL DOWN FOR ARTICLEFull terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdfThis article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction,re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expresslyforbidden.The publisher does not give any warranty express or implied or make any representation that the contents will becomplete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should beindependently verified with primary sources. The publisher shall not be liable…

AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions

Chunhui Gu, Chen Sun, David A. Ross +9 more

cs.CV · May 2017 · cited 1,240× · arXiv ↗ · more like this ·

This paper introduces a video dataset of spatio-temporally localized Atomic Visual Actions (AVA). The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1.58M action labels with multiple labels per person occurring frequently. The key characteristics of our dataset are: (1) the definition of atomic visual actions, rather than composite actions; (2) precise spatio-temporal annotations with possibly multiple annotations for each person; (3) exhaustive annotation of these atomic actions over 15-minute video clips; (4) people temporally linked across consecutive segments; and (5) using movies to gather a varied set of action representations. This departs from existing datasets for spatio-temporal action recognition, which typically provide sparse annotations for composite actions in short video…

Multi-scale Attributed Node Embedding

Benedek Rozemberczki, Carl Allen, Rik Sarkar

cs.LG · September 2019 · cited 1,159× · arXiv ↗ · more like this ·

We present network embedding algorithms that capture information about a node from the local distribution over node attributes around it, as observed over random walks following an approach similar to Skip-gram. Observations from neighborhoods of different sizes are either pooled (AE) or encoded distinctly in a multi-scale approach (MUSAE). Capturing attribute-neighborhood relationships over multiple scales is useful for a diverse range of applications, including latent feature identification across disconnected networks with similar attributes. We prove theoretically that matrices of node-feature pointwise mutual information are implicitly factorized by the embeddings. Experiments show that our algorithms are robust, computationally efficient and outperform comparable models on social networks and web graphs.