arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Yanping

2,810 papers · 1753–2026

1967: 01968: 01969: 01970: 01971: 01972: 01973: 01974: 01975: 01976: 11977: 01978: 01979: 01980: 01981: 01982: 01983: 01984: 01985: 01986: 11987: 01988: 21989: 01990: 21991: 31992: 01993: 11994: 41995: 11996: 11997: 41998: 11999: 62000: 172001: 122002: 322003: 182004: 502005: 702006: 862007: 942008: 952009: 1582010: 1432011: 1082012: 962013: 1002014: 822015: 992016: 742017: 772018: 1002019: 1042020: 1322021: 1582022: 1752023: 1932024: 2132025: 1912026: 101
1967peak 213 in 20242026

Frequent co-authors

Papers

showing 1–25 of 2,810 RSS

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI, Daya Guo, Dejian Yang +197 more

cs.CL · · cited 5,800× · arXiv ↗ · more like this ·

General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) and chain-of-thought prompting, have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models' capabilities are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and…

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre +32 more

cs.LG · · cited 4,465× · arXiv ↗ · more like this ·

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve…

Regularized Evolution for Image Classifier Architecture Search

Esteban Real, Alok Aggarwal, Yanping Huang +1 more

cs.NE · · cited 3,454× · arXiv ↗ · more like this ·

The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically. Although evolutionary algorithms have been repeatedly applied to neural network topologies, the image classifiers thus discovered have remained inferior to human-crafted ones. Here, we evolve an image classifier---AmoebaNet-A---that surpasses hand-designs for the first time. To do this, we modify the tournament selection evolutionary algorithm by introducing an age property to favor the younger genotypes. Matching size, AmoebaNet-A has comparable accuracy to current state-of-the-art ImageNet models discovered with more complex architecture-search methods. Scaled to larger size, AmoebaNet-A sets a new state-of-the-art 83.9% / 96.6% top-5 ImageNet accuracy. In a controlled comparison against a well known reinforcement learning algorithm, we give…

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu +6 more

cs.CL · · cited 2,424× · arXiv ↗ · more like this ·

Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data and compute. Although this trend of scaling is affirmed to be a sure-fire approach for better model quality, there are challenges on the path such as the computation cost, ease of programming, and efficient implementation on parallel devices. GShard is a module composed of a set of lightweight annotation APIs and an extension to the XLA compiler. It provides an elegant way to express a wide range of parallel computation patterns with minimal changes to the existing model code. GShard enabled us to scale up multilingual neural machine translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600 billion parameters using automatic sharding. We demonstrate that such a giant model can efficiently be trained on 2048 TPU…

LaMDA: Language Models for Dialog Applications

Romal Thoppilan, Daniel De Freitas, Jamie Hall +57 more

cs.CL · · cited 1,922× · arXiv ↗ · more like this ·

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier…

PaLM 2 Technical Report

Rohan Anil, Andrew M. Dai, Orhan Firat +125 more

cs.CL · · cited 1,563× · arXiv ↗ · more like this ·

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables…

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI, Aixin Liu, Bei Feng +154 more

cs.CL · · cited 1,487× · arXiv ↗ · more like this ·

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and…

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Nan Du, Yanping Huang, Andrew M. Dai +24 more

cs.CL · · cited 1,338× · arXiv ↗ · more like this ·

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP…

Associations of healthy lifestyle and socioeconomic status with mortality and incident cardiovascular disease: two prospective cohort studies

Yanbo Zhang, Chen Chen, Xiong‐Fei Pan +5 more

Health · BMJ · · cited 944× · DOI ↗ · more like this ·

OBJECTIVE: To examine whether overall lifestyles mediate associations of socioeconomic status (SES) with mortality and incident cardiovascular disease (CVD) and the extent of interaction or joint relations of lifestyles and SES with health outcomes. DESIGN: Population based cohort study. SETTING: US National Health and Nutrition Examination Survey (US NHANES, 1988-94 and 1999-2014) and UK Biobank. PARTICIPANTS: 44 462 US adults aged 20 years or older and 399 537 UK adults aged 37-73 years. EXPOSURES: SES was derived by latent class analysis using family income, occupation or employment status, education level, and health insurance (US NHANES only), and three levels (low, medium, and high) were defined according to item response probabilities. A healthy lifestyle score was constructed using information on never smoking, no heavy alcohol consumption (women ≤1 drink/day; men ≤2 drinks/day…

Impact of Healthy Lifestyle Factors on Life Expectancies in the US Population

Yanping Li, An Pan, Dong D. Wang +8 more

Health · Circulation · · cited 942× · DOI ↗ · more like this ·

BACKGROUND: Americans have a shorter life expectancy compared with residents of almost all other high-income countries. We aim to estimate the impact of lifestyle factors on premature mortality and life expectancy in the US population. METHODS: , ≥30 min/d of moderate to vigorous physical activity, moderate alcohol intake, and a high diet quality score (upper 40%), and estimated hazard ratios for the association of total lifestyle score (0-5 scale) with mortality. We used data from the NHANES (National Health and Nutrition Examination Surveys; 2013-2014) to estimate the distribution of the lifestyle score and the US Centers for Disease Control and Prevention WONDER database to derive the age-specific death rates of Americans. We applied the life table method to estimate life expectancy by levels of the lifestyle score. RESULTS: During up to 34 years of follow-up, we documented 42 167…

Mixture-of-Experts with Expert Choice Routing

Yanqi Zhou, Tao Lei, Hanxiao Liu +7 more

cs.LG · · cited 923× · arXiv ↗ · more like this ·

Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational…

Gemini: A Family of Highly Capable Multimodal Models

Gemini Team, Rohan Anil, Sebastian Borgeaud +1348 more

cs.CL · · cited 816× · arXiv ↗ · more like this ·

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models…

Brief Report: Increased Addictive Internet and Substance Use Behavior During the COVID‐19 Pandemic in China

Yan Sun, Yangyang Li, Yanping Bao +7 more

Sociology and Political Science · American Journal on Addictions · · cited 500× · DOI ↗ · more like this ·

BACKGROUND AND OBJECTIVES: The COVID-19 pandemic and control measures may have increased the risk of abusing addictive substances as well as addictive behaviors. METHODS: We present an initial online survey in 6416 Chinese about the relation between the COVID-19 pandemic and addictive behavior in China. RESULTS: During the COVID-19 pandemic, 46.8% of the subjects reported increased dependence on internet use, and 16.6% had longer hours of internet use. The prevalence (4.3%) of severe internet dependence rose up to 23% than that (3.5%) before the COVID-19 pandemic occurred, and their dependence degree rose 20 times more often than being declined (60% vs 3%). Relapses to abuse from alcohol and smoking abstinence were relatively common at 19% and 25%, respectively. Similarly, 32% of regular alcohol drinkers and 20% of regular smokers increased their usage amount during the pandemic.…

ST-MoE: Designing Stable and Transferable Sparse Expert Models

Barret Zoph, Irwan Bello, Sameer Kumar +5 more

cs.CL · · cited 472× · arXiv ↗ · more like this ·

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book…

Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

Zhao Chen, Jiquan Ngiam, Yanping Huang +4 more

cs.LG · · cited 313× · arXiv ↗ · more like this ·

The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these multiple updates can impede optimal training by pulling the model in conflicting directions. We present Gradient Sign Dropout (GradDrop), a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. GradDrop is implemented as a simple deep layer that can be used in any deep net and synergizes with other gradient balancing approaches. We show that GradDrop outperforms the state-of-the-art multiloss methods within traditional multitask and transfer learning settings, and we discuss how GradDrop reveals links between optimal multiloss training and gradient stochasticity.

AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Zhuohan Li, Lianmin Zheng, Yinmin Zhong +8 more

cs.LG · · cited 292× · arXiv ↗ · more like this ·

Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher…

Sampled-Data Based Consensus of Continuous-Time Multi-Agent Systems With Time-Varying Topology

Yanping Gao, Long Wang

Computer Networks and Communications · IEEE Transactions on Automatic Control · · cited 287× · DOI ↗ · more like this ·

This technical note studies consensus problems of multiple agents with continuous-time second-order dynamics, where each agent can obtain its positions and velocities relative to its neighbors only at sampling instants. It is assumed that the sampling period of each agent is independent of the others' and the interaction topology among agents is time-varying, where the associated direct graphs may not have spanning trees. If the union graph of all direct graphs has a spanning tree, then there exist controller gains and sampling periods such that consensus is reached. Moreover, two approaches are presented to design such controller gains and sampling periods. Simulations are performed to validate the theoretical results.

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Lianmin Zheng, Zhuohan Li, Hao Zhang +9 more

cs.LG · · cited 283× · arXiv ↗ · more like this ·

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations. They do not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive efficient parallel execution plans at each parallelism level. Alpa implements an efficient runtime to orchestrate the two-level parallel execution on…

Convergence analysis of the Jacobi spectral-collocation methods for Volterra integral equations with a weakly singular kernel

Yanping Chen, Tao Tang

Modeling and Simulation · Mathematics of Computation · · cited 255× · DOI ↗ · more like this ·

In this paper, a Jacobi-collocation spectral method is developed for Volterra integral equations of the second kind with a weakly singular kernel. We use some function transformations and variable transformations to change the equation into a new Volterra integral equation defined on the standard interval [-1, 1], so that the solution of the new equation possesses better regularity and the Jacobi orthogonal polynomial theory can be applied conveniently. In order to obtain high-order accuracy for the approximation, the integral term in the resulting equation is approximated by using Jacobi spectral quadrature rules. The convergence analysis of this novel method is based on the Lebesgue constants corresponding to the Lagrange interpolation polynomials, polynomial approximation theory for orthogonal polynomials and operator theory. The spectral rate of convergence for the proposed method…

Sustainable Work Performance: The Roles of Workplace Violence and Occupational Stress

Samma Faiz Rasool, Mansi Wang, Yanping Zhang +1 more

Sociology and Political Science · International Journal of Environmental Research and Public Health · · cited 238× · DOI ↗ · more like this ·

The purpose of this study is to analyze the relationships between workplace violence, occupational stress, and sustainable work performance. Multiple dimensions of workplace violence (harassment, mobbing, ostracism, and stalking) were used in this study. A questionnaire survey was used, composed of 48 items with a 5-point Likert scale (1, strongly disagree, to 5, strongly agree). Data were collected from 15 hospitals in the vicinity of Karachi, Lahore, and Islamabad, Pakistan. The target population of this study consisted of doctors, nurses, and paramedical staff. We distributed 500 questionnaires among the target population. In total, 345 usable questionnaires were returned, resulting in a response rate of 69%. Partial least squares structural equation modeling was used to test the direct and indirect effects. The results of this study highlight that in both direct and indirect…

Dynamic Time Warping Averaging of Time Series Allows Faster and More Accurate Classification

François Petitjean, Germain Forestier, Geoffrey I. Webb +3 more

Signal Processing · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 237× · DOI ↗ · more like this ·

Recent years have seen significant progress in improving both the efficiency and effectiveness of time series classification. However, because the best solution is typically the Nearest Neighbor algorithm with the relatively expensive Dynamic Time Warping as the distance measure, successful deployments on resource constrained devices remain elusive. Moreover, the recent explosion of interest in wearable devices, which typically have limited computational resources, has created a growing need for very efficient classification algorithms. A commonly used technique to glean the benefits of the Nearest Neighbor algorithm, without inheriting its undesirable time complexity, is to use the Nearest Centroid algorithm. However, because of the unique properties of (most) time series data, the centroid typically does not resemble any of the instances, an unintuitive and underappreciated fact. In…

Internet Addiction Increases in the General Population During COVID‐19: Evidence From China

Yangyang Li, Yan Sun, Shiqiu Meng +9 more

Sociology and Political Science · American Journal on Addictions · · cited 219× · DOI ↗ · more like this ·

BACKGROUND AND OBJECTIVES: COVID-19-related quarantine and stress have likely escalated the crisis of Internet addiction. This study aimed to determine the impact of the COVID-19 pandemic on Internet use and related risk factors among the general public in China. METHODS: A large-sample cross-sectional online survey was conducted from March 24 to April 30, 2020, in China, and 20,472 participants completed the survey. We investigated the prevalence and severity of Internet addiction based on the Internet Addiction Test (IAT), and explored the risk factors related to increases in time spent on Internet use and severity of Internet addiction, as well as severe Internet addiction. RESULTS: The overall prevalence of Internet addiction was 36.7% among the general population during the pandemic, and that of severe Internet addiction was 2.8%, according to IAT scores. Time spent on recreational…

Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling

Jonathan Shen, Patrick Nguyen, Yonghui Wu +88 more

cs.LG · · cited 215× · arXiv ↗ · more like this ·

Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models. Lingvo models are composed of modular building blocks that are flexible and easily extensible, and experiment configurations are centralized and highly customizable. Distributed training and quantized inference are supported directly within the framework, and it contains existing implementations of a large number of utilities, helper functions, and the newest research ideas. Lingvo has been used in collaboration by dozens of researchers in more than 20 papers over the last two years. This document outlines the underlying design of Lingvo and serves as an introduction to the various pieces of the framework, while also offering examples of advanced features that showcase the capabilities of the framework.

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Yu Zhang, Daniel S. Park, Wei Han +23 more

eess.AS · · cited 212× · arXiv ↗ · more like this ·

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of…

FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing

Yuren Cong, Mengmeng Xu, Christian Simon +7 more

cs.CV · · cited 206× · arXiv ↗ · more like this ·

Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited video are visually consistent. Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention. Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch and therefore cause inconsistency in the edited video. In this paper, for the first time, we introduce optical flow into the attention module in the diffusion model's U-Net to address the inconsistency issue for text-to-video editing. Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other in the attention module, thus improving the visual…