arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Devlin

1,797 papers · 1858–2026

1967: 11968: 11969: 11970: 11971: 11972: 31973: 11974: 21975: 21976: 11977: 41978: 41979: 81980: 31981: 31982: 21983: 11984: 131985: 21986: 71987: 31988: 51989: 51990: 71991: 51992: 71993: 71994: 121995: 211996: 131997: 151998: 231999: 222000: 192001: 152002: 272003: 152004: 262005: 352006: 342007: 562008: 492009: 642010: 712011: 622012: 572013: 632014: 652015: 642016: 852017: 772018: 682019: 812020: 842021: 992022: 472023: 802024: 832025: 792026: 72
1967peak 99 in 20212026

Frequent co-authors

Papers

showing 1–25 of 1,797 RSS

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee +1 more

cs.CL · · cited 121,876× · arXiv ↗ · more like this ·

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question…

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin +64 more

cs.CL · · cited 8,437× · arXiv ↗ · more like this ·

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite…

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre +32 more

cs.LG · · cited 4,465× · arXiv ↗ · more like this ·

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve…

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team, Petko Georgiev, Ving Ian Lei +1134 more

cs.CL · · cited 4,033× · arXiv ↗ · more like this ·

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's…

Natural Questions: A Benchmark for Question Answering Research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield +9 more

Artificial Intelligence · Transactions of the Association for Computational Linguistics · · cited 1,994× · DOI ↗ · more like this ·

We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating…

PaLM 2 Technical Report

Rohan Anil, Andrew M. Dai, Orhan Firat +125 more

cs.CL · · cited 1,563× · arXiv ↗ · more like this ·

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables…

Gemini: A Family of Highly Capable Multimodal Models

Gemini Team, Rohan Anil, Sebastian Borgeaud +1348 more

cs.CL · · cited 816× · arXiv ↗ · more like this ·

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models…

Efficiently Scaling Transformer Inference

Reiner Pope, Sholto Douglas, Aakanksha Chowdhery +7 more

cs.LG · · cited 787× · arXiv ↗ · more like this ·

We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Better understanding of the engineering tradeoffs for inference for large Transformer-based models is important as use cases of these models are growing rapidly throughout application areas. We develop a simple analytical model for inference efficiency to select the best multi-dimensional partitioning techniques optimized for TPU v4 slices based on the application requirements. We combine these with a suite of low-level optimizations to achieve a new Pareto frontier on the latency and model FLOPS utilization (MFU) tradeoffs on 500B+ parameter models that outperforms the FasterTransformer suite of benchmarks. We further show that with appropriate partitioning, the lower memory requirements of…

Visual Storytelling

Ting-Hao, Huang, Francis Ferraro +13 more

cs.CL · · cited 550× · arXiv ↗ · more like this ·

We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling. The first release of this dataset, SIND v.1, includes 81,743 unique photos in 20,211 sequences, aligned to both descriptive (caption) and story language. We establish several strong baselines for the storytelling task, and motivate an automatic metric to benchmark progress. Modelling concrete description as well as figurative and social language, as provided in this dataset and the storytelling task, has the potential to move artificial intelligence from basic understandings of typical visual scenes towards more and more human-like understanding of grounded event structure and subjective expression.

A survey of literature on the teaching of introductory programming

Arnold Pears, Stephen B. Seidman, Lauri Malmi +5 more

Computer Science Applications · ACM SIGCSE Bulletin · · cited 481× · DOI ↗ · more like this ·

Three decades of active research on the teaching of introductory programming has had limited effect on classroom practice. Although relevant research exists across several disciplines including education and cognitive science, disciplinary differences have made this material inaccessible to many computing educators. Furthermore, computer science instructors have not had access to a comprehensive survey of research in this area. This paper collects and classifies this literature, identifies important work and mediates it to computing educators and professional bodies. We identify research that gives well-supported advice to computing academics teaching introductory programming. Limitations and areas of incomplete coverage of existing research efforts are also identified. The analysis applies publication and research quality metrics developed by a previous ITiCSE working group [74].

RobustFill: Neural Program Learning under Noisy I/O

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju +3 more

cs.AI · · cited 425× · arXiv ↗ · more like this ·

The problem of automatically generating a computer program from some specification has been studied since the early days of AI. Recently, two competing approaches for automatic program learning have received significant attention: (1) neural program synthesis, where a neural network is conditioned on input/output (I/O) examples and learns to generate a program, and (2) neural program induction, where a neural network generates new outputs directly using a latent program representation. Here, for the first time, we directly compare both approaches on a large-scale, real-world learning task. We additionally contrast to rule-based program synthesis, which uses hand-crafted semantics to guide the program generation. Our neural models use a modified attention RNN to allow encoding of variable-sized sets of I/O pairs. Our best synthesis model achieves 92% accuracy on a real-world test set…

Robust estimation and outlier detection with correlation coefficients

Susan J. Devlin, R. Gnanadesikan, J. R. Kettenring

Statistics and Probability · Biometrika · · cited 386× · DOI ↗ · more like this ·

Two graphical methods are proposed for identifying bivariate observations that may unduly influence the sample correlation coefficient. Secondly, robust estimators of correlation are developed and a Monte Carlo comparative study is made of these and other well-known estimators. Also considered are methods for developing positive-definite estimates of correlation matrices and extensions of robustness to other problems such as regression are mentioned.

Robust Estimation of Dispersion Matrices and Principal Components

Susan J. Devlin, R. Gnanadesikan, J. R. Kettenring

Statistics and Probability · Journal of the American Statistical Association · · cited 378× · DOI ↗ · more like this ·

Abstract This paper uses Monte Carlo methods to compare the performances of several robust procedures for estimating a correlation matrix and its principal components. The estimators are formed either from separate bivariate analyses or by simultaneous manipulation of all variables by using techniques such as multivariate trimming and M-estimation. The M-estimators stand up exceptionally well. They and the multivariate trimming procedure are especially effective at estimating the principal components, including a near singularity. However, the M-estimators can break down relatively easily when the dimensionality is large and the outliers are asymmetric. With missing data, the element-wise approach becomes more attractive.

The criteria of effective teaching in a changing higher education context

Marcia Devlin, Gayani Samarawickrema

Education · Higher Education Research & Development · · cited 339× · DOI ↗ · more like this ·

The criteria of effective teaching in higher education are understood to comprise particular skills and practices applied within particular contexts. Drawing on the literature and using Australia’s understanding of effective teaching, this paper examines the notion of effective teaching. The paper specifically compares dimensions derived from robust research and psychometric processes with the Australian Learning and Teaching Council’s criteria for effective teaching and observes the criteria of effective teaching in higher education to have evolved. While the paper suggests some areas in which future considerations of the notion of effective teaching might usefully focus, it also argues that context is critical and that it is subject to continuous and multiple changes imposed by forces from within and outside universities. The paper maintains that our collective understanding of…

Generating Natural Questions About an Image

Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin +3 more

cs.CL · · cited 317× · arXiv ↗ · more like this ·

There has been an explosion of work in the vision & language community during the past few years from image captioning to video transcription, and answering questions about images. These tasks have focused on literal descriptions of the image. To move beyond the literal, we choose to explore how questions about an image are often directed at commonsense inference and the abstract events evoked by objects in the image. In this paper, we introduce the novel task of Visual Question Generation (VQG), where the system is tasked with asking a natural and engaging question when shown an image. We provide three datasets which cover a variety of images from object-centric to event-centric, with considerably more abstract training data than provided to state-of-the-art captioning systems thus far. We train and test several generative and retrieval models to tackle the task of VQG. Evaluation…

Universal Neural Machine Translation for Extremely Low Resource Languages

Jiatao Gu, Hany Hassan, Jacob Devlin +1 more

cs.CL · · cited 307× · arXiv ↗ · more like this ·

In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multilingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multilingual training and…

Zero-Shot Entity Linking by Reading Entity Descriptions

Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee +3 more

cs.CL · · cited 294× · arXiv ↗ · more like this ·

We present the zero-shot entity linking task, where mentions must be linked to unseen entities without in-domain labeled data. The goal is to enable robust transfer to highly specialized domains, and so no metadata or alias tables are assumed. In this setting, entities are only identified by text descriptions, and models must rely strictly on language understanding to resolve the new entities. First, we show that strong reading comprehension models pre-trained on large unlabeled data can be used to generalize to unseen entities. Second, we propose a simple and effective adaptive pre-training strategy, which we term domain-adaptive pre-training (DAP), to address the domain shift problem associated with linking unseen entities in a new domain. We present experiments on a new dataset that we construct for this task and show that DAP improves over strong pre-training baselines, including…

Language Models for Image Captioning: The Quirks and What Works

Jacob Devlin, Hao Cheng, Hao Fang +5 more

cs.CL · · cited 293× · arXiv ↗ · more like this ·

Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO…

Synthetic QA Corpora Generation with Roundtrip Consistency

Chris Alberti, Daniel Andor, Emily Pitler +2 more

cs.CL · · cited 281× · arXiv ↗ · more like this ·

We introduce a novel method of generating synthetic question answering corpora by combining models of question generation and answer extraction, and by filtering the results to ensure roundtrip consistency. By pretraining on the resulting corpora we obtain significant improvements on SQuAD2 and NQ, establishing a new state-of-the-art on the latter. Our synthetic data generation models, for both question generation and answer extraction, can be fully reproduced by finetuning a publicly available BERT model on the extractive subsets of SQuAD2 and NQ. We also describe a more powerful variant that does full sequence-to-sequence pretraining for question generation, obtaining exact match and F1 at less than 0.1% and 0.4% from human performance on SQuAD2.

The Effect of Increased Women's Representation in Parliament: The Case of Rwanda

Claire Devlin, Robert Elgie

Gender Studies · Parliamentary Affairs · · cited 267× · DOI ↗ · more like this ·

There is an increasing amount of work on whether a higher level of women's representation in parliament leads to a different style of parliamentary politics. To date, most studies have focused on Western cases, and the results have been mixed. Women add new dimensions to the policy agenda, but there is little evidence that increased women's representation changes policy outputs. The little work that has been conducted outside the Western context confirms the mixed nature of these findings. In sub-Saharan Africa, women have added issues such as HIV/AIDS and property rights to the policy agenda, but there is little evidence to suggest that increased women's representation has altered policy outcomes. In this article, we examine the case of Rwanda, which now has the highest level of women's representation in parliament in the world at 48.75 per cent. Based on face-to-face interviews with…

In their own words: a qualitative study of the reasons Australian university students plagiarize

Marcia Devlin, Kathleen Gray

Safety Research · Higher Education Research & Development · · cited 262× · DOI ↗ · more like this ·

Abstract The ways in which universities and individual academics attempt to deter and respond to student plagiarism may be based on untested assumptions about particular or primary reasons for this behaviour. Using a series of group interviews, this qualitative study gathered the views of 56 Australian university students on the possible reasons for plagiarism within their institution. The results indicate a wide and disparate range of possible contributing reasons for plagiarism, including: institutional admission criteria; student understanding of plagiarism; poor academic skills; a range of teaching and learning factors; personality factors; and external pressures. These findings are compared with other findings about reasons for student plagiarism in Australasia. The implications of these findings are considered for universities and individual academics seeking to better engage with…

Articular area responses to mechanical loading: effects of exercise, age, and skeletal location

Daniel E. Lieberman, Maureen J. Devlin, Osbjorn M. Pearson

Anthropology · American Journal of Physical Anthropology · · cited 248× · DOI ↗ · more like this ·

How reliable are reconstructions of body mass and joint function based on articular surface areas? While the dynamic relationship between mechanical loading and cross-sectional geometry in long bones is well-established, the effect of loading on the subchondral articular surface area of epiphyses (hereafter, articular surface area, or ASA) has not been experimentally tested. The degree to which ASA can change in size and shape is important, because articular dimensions are frequently used to estimate body mass and positional behavior in fossil species. This study tests the hypothesis that mechanical loading influences ASA by comparing epiphyses of exercised and sedentary sheep from three age categories: juvenile, subadult, and adult (n = 44). ASA was measured on latex molds of subchondral articular surfaces of 10 epiphyses from each sheep. Areas were standardized by body mass, and…

Bridging socio-cultural incongruity: conceptualising the success of students from low socio-economic status backgrounds in Australian higher education

Marcia Devlin

Education · Studies in Higher Education · · cited 235× · DOI ↗ · more like this ·

This article examines the conceptual frames that might be used to consider the success and achievement of students from low socio-economic status in Australian higher education. Based on an examination of key literature from Australia, New Zealand, the United Kingdom and North America, it is argued that Australia should avoid adopting either a deficit conception of students from low socio-economic backgrounds or a deficit conception of the institutions into which they will move. Further, rather than it being the primary responsibility of the student or of the institution to change to ensure the success of these students, it is argued that the adjustments necessary to ensure achievement for students from low socio-economic backgrounds in Australian higher education would be most usefully conceptualised as a ‘joint venture’ toward bridging socio-cultural incongruity.

Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis

Rudy Bunel, Matthew Hausknecht, Jacob Devlin +2 more

cs.LG · · cited 233× · arXiv ↗ · more like this ·

Program synthesis is the task of automatically generating a program consistent with a specification. Recent years have seen proposal of a number of neural approaches for program synthesis, many of which adopt a sequence generation paradigm similar to neural machine translation, in which sequence-to-sequence models are trained to maximize the likelihood of known reference programs. While achieving impressive results, this strategy has two key limitations. First, it ignores Program Aliasing: the fact that many different programs may satisfy a given specification (especially with incomplete specifications such as a few input-output examples). By maximizing the likelihood of only a single reference program, it penalizes many semantically correct programs, which can adversely affect the synthesizer performance. Second, this strategy overlooks the fact that programs have a strict syntax that…