arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Dean

6,984 papers · 1874–2026

1967: 101968: 181969: 211970: 191971: 191972: 191973: 151974: 151975: 141976: 301977: 231978: 231979: 231980: 241981: 301982: 241983: 371984: 371985: 331986: 421987: 491988: 441989: 571990: 551991: 601992: 671993: 641994: 781995: 811996: 841997: 831998: 971999: 1222000: 1082001: 1192002: 1752003: 1352004: 1792005: 2262006: 1962007: 2042008: 2002009: 2442010: 2392011: 2492012: 2322013: 2722014: 2662015: 2732016: 1662017: 1462018: 1552019: 1672020: 1952021: 2152022: 2172023: 2012024: 1972025: 2182026: 250
1967peak 273 in 20152026

Frequent co-authors

Papers

showing 1–25 of 6,984 RSS

Distributed Representations of Words and Phrases and their Compositionality

Tomas Mikolov, Ilya Sutskever, Kai Chen +2 more

cs.CL · · cited 35,733× · arXiv ↗ · more like this ·

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector…

Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, Kai Chen, Greg Corrado +1 more

cs.CL · · cited 35,169× · arXiv ↗ · more like this ·

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.

Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean

stat.ML · · cited 26,484× · arXiv ↗ · more like this ·

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble…

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Martín Abadi, Ashish Agarwal, Paul Barham +37 more

cs.DC · · cited 11,866× · arXiv ↗ · more like this ·

TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction…

Applied Multivariate Statistical Analysis.

Andrea Johnson, Dean W. Wichern

Statistics and Probability · Biometrics · · cited 11,439× · DOI ↗ · more like this ·

(NOTE: Each chapter begins with an Introduction, and concludes with Exercises and References.) I. GETTING STARTED. 1. Aspects of Multivariate Analysis. Applications of Multivariate Techniques. The Organization of Data. Data Displays and Pictorial Representations. Distance. Final Comments. 2. Matrix Algebra and Random Vectors. Some Basics of Matrix and Vector Algebra. Positive Definite Matrices. A Square-Root Matrix. Random Vectors and Matrices. Mean Vectors and Covariance Matrices. Matrix Inequalities and Maximization. Supplement 2A Vectors and Matrices: Basic Concepts. 3. Sample Geometry and Random Sampling. The Geometry of the Sample. Random Samples and the Expected Values of the Sample Mean and Covariance Matrix. Generalized Variance. Sample Mean, Covariance, and Correlation as Matrix Operations. Sample Values of Linear Combinations of Variables. 4. The Multivariate Normal…

TensorFlow: A system for large-scale machine learning

Martín Abadi, Paul Barham, Jianmin Chen +19 more

cs.DC · · cited 8,825× · arXiv ↗ · more like this ·

TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. TensorFlow uses dataflow graphs to represent computation, shared state, and the operations that mutate that state. It maps the nodes of a dataflow graph across many machines in a cluster, and within a machine across multiple computational devices, including multicore CPUs, general-purpose GPUs, and custom designed ASICs known as Tensor Processing Units (TPUs). This architecture gives flexibility to the application developer: whereas in previous "parameter server" designs the management of shared state is built into the system, TensorFlow enables developers to experiment with novel optimizations and training algorithms. TensorFlow supports a variety of applications, with particularly strong support for training and inference on deep neural networks. Several Google services use…

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin +64 more

cs.CL · · cited 8,437× · arXiv ↗ · more like this ·

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite…

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Yonghui Wu, Mike Schuster, Zhifeng Chen +28 more

cs.CL · · cited 7,404× · arXiv ↗ · more like this ·

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To…

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz +4 more

cs.LG · · cited 5,777× · arXiv ↗ · more like this ·

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language…

In-Datacenter Performance Analysis of a Tensor Processing Unit

Norman P. Jouppi, Cliff Young, Nishant Patil +72 more

cs.AR · · cited 5,472× · arXiv ↗ · more like this ·

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU…

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre +32 more

cs.LG · · cited 4,465× · arXiv ↗ · more like this ·

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve…

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team, Petko Georgiev, Ving Ian Lei +1134 more

cs.CL · · cited 4,033× · arXiv ↗ · more like this ·

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's…

Emergent Abilities of Large Language Models

Jason Wei, Yi Tay, Rishi Bommasani +13 more

cs.CL · · cited 3,867× · arXiv ↗ · more like this ·

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

Bigtable

Fay W. Chang, Jay B. Dean, Sanjay Ghemawat +6 more

Computer Networks and Communications · ACM Transactions on Computer Systems · · cited 3,432× · DOI ↗ · more like this ·

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this article, we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

Staged Authenticity: Arrangements of Social Space in Tourist Settings

Dean MacCannell

Sociology and Political Science · American Journal of Sociology · · cited 3,206× · DOI ↗ · more like this ·

The problem of false consciousness and its relationship to the social structure of tourist establishments is analyzed. Accounts of travelers are examined in terms of Erving Goffman's front versus back distinction. It is found that tourists try to enter back regions of the places they visit because these regions are associated with intimacy of relations and authenticity of experiences. It is also found that tourist settings are arrenged to produce the impression that a back region has been entered even when this is not the case. In tourist settings, between the front and the back there is a series of special spaces designed to accommodate tourists and to support their beliefs in the authenticity of their experiences. Goffman's front-back dichotomy is shown to be ideal poles of a continuum, or a variable.

Efficient Neural Architecture Search via Parameter Sharing

Hieu Pham, Melody Y. Guan, Barret Zoph +2 more

cs.LG · · cited 3,093× · arXiv ↗ · more like this ·

We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Thanks to parameter sharing between child models, ENAS is fast: it delivers strong empirical performances using much fewer GPU-hours than all existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On the Penn Treebank dataset, ENAS discovers a novel architecture that achieves a test perplexity of 55.8, establishing a new…

Scalable and accurate deep learning for electronic health records

Alvin Rajkomar, Eyal Oren, Kai Chen +31 more

cs.CY · · cited 2,982× · arXiv ↗ · more like this ·

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire, raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two U.S. academic medical centers with 216,221 adult patients hospitalized for at least 24 hours. In the sequential format we propose, this…

DrugBank: a knowledgebase for drugs, drug actions and drug targets

David S. Wishart, Craig Knox, An Chi Guo +5 more

Computational Theory and Mathematics · Nucleic Acids Research · · cited 2,917× · DOI ↗ · more like this ·

DrugBank is a richly annotated resource that combines detailed drug data with comprehensive drug target and drug action information. Since its first release in 2006, DrugBank has been widely used to facilitate in silico drug target discovery, drug design, drug docking or screening, drug metabolism prediction, drug interaction prediction and general pharmaceutical education. The latest version of DrugBank (release 2.0) has been expanded significantly over the previous release. With approximately 4900 drug entries, it now contains 60% more FDA-approved small molecule and biotech drugs including 10% more 'experimental' drugs. Significantly, more protein target data has also been added to the database, with the latest version of DrugBank containing three times as many non-redundant protein or drug target sequences as before (1565 versus 524). Each DrugCard entry now contains more than 100…

Gemma 2: Improving Open Language Models at a Practical Size

Gemma Team, Morgane Riviere, Shreya Pathak +195 more

cs.CL · · cited 2,513× · arXiv ↗ · more like this ·

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

OWL Web Ontology Language - Reference

Mike Dean, A.T. Schreiber, Sean Bechofer +6 more

Artificial Intelligence · Data Archiving and Networked Services (DANS) · · cited 2,392× · OpenAlex ↗ · more like this ·

The {W}eb {O}ntology {L}anguage {OWL} is a semantic markup language for publishing and sharing ontologies on the {W}orld {W}ide {W}eb. {OWL} is developed as a vocabulary extension of {RDF} (the {R}esource {D}escription {F}ramework) and is derived from the {DAML}+{OIL} {W}eb {O}ntology {L}anguage. {T}his document contains a structured informal description of the full set of {OWL} language constructs and is meant to serve as a reference for {OWL} users who want to construct {OWL} ontologies.

Building high-level features using large scale unsupervised learning

Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga +5 more

cs.LG · · cited 2,330× · arXiv ↗ · more like this ·

We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and…

Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Melvin Johnson, Mike Schuster, Quoc V. Le +9 more

cs.CL · · cited 2,264× · arXiv ↗ · more like this ·

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no change in the model architecture from our base system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes encoder, decoder and attention, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. Our method often improves the translation quality of all involved language pairs, even while keeping the total number of model parameters constant. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for…

geomorph: an<scp>r</scp>package for the collection and analysis of geometric morphometric shape data

Dean C. Adams, Erik Otárola‐Castillo

Geometry and Topology · Methods in Ecology and Evolution · · cited 2,186× · DOI ↗ · more like this ·

Summary Many ecological and evolutionary studies seek to explain patterns of shape variation and its covariation with other variables. Geometric morphometrics is often used for this purpose, where a set of shape variables are obtained from landmark coordinates following a P rocrustes superimposition. We introduce geomorph: a software package for performing geometric morphometric shape analysis in the r statistical computing environment. Geomorph provides routines for all stages of landmark‐based geometric morphometric analyses in two and three‐dimensions. It is an open source package to read, manipulate, and digitize landmark data, generate shape variables via P rocrustes analysis for points, curves and surfaces, perform statistical analyses of shape variation and covariation, and to provide graphical depictions of shapes and patterns of shape variation. An important contribution of…

Geometric morphometrics: Ten years of progress following the ‘revolution’

Dean C. Adams, F. James Rohlf, Dennis E. Slice

Geometry and Topology · Italian Journal of Zoology · · cited 2,181× · DOI ↗ · more like this ·

Abstract The analysis of shape is a fundamental part of much biological research. As the field of statistics developed, so have the sophistication of the analysis of these types of data. This lead to multivariate morphometrics in which suites of measurements were analyzed together using canonical variates analysis, principal components analysis, and related methods. In the 1980s, a fundamental change began in the nature of the data gathered and analyzed. This change focused on the coordinates of landmarks and the geometric information about their relative positions. As a by‐product of such an approach, results of multivariate analyses could be visualized as configurations of landmarks back in the original space of the organism rather than only as statistical scatter plots. This new approach, called “geometric morphometrics”;, had benefits that lead Rohlf and Marcus (1993) to proclaim a…

DeViSE: A Deep Visual-Semantic Embedding Model

Andrea Frome, Greg S. Corrado, Jon Shlens +4 more

Artificial Intelligence · Neural Information Processing Systems · · cited 2,069× · OpenAlex ↗ · more like this ·

Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories. This limitation is in part due to the increasing difficulty of acquiring sufficient training data in the form of labeled images as the number of object categories grows. One remedy is to leverage data from other sources – such as text data – both to train visual models and to constrain their pre-dictions. In this paper we present a new deep visual-semantic embedding model trained to identify visual objects using both labeled image data as well as seman-tic information gleaned from unannotated text. We demonstrate that this model matches state-of-the-art performance on the 1000-class ImageNet object recogni-tion challenge while making more semantically reasonable errors, and also show that the semantic information can be exploited to make predictions about tens of…