arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Yujun Lin

55 papers · 2007–2026

2007: 12008: 02009: 02010: 02011: 12012: 12013: 02014: 02015: 12016: 22017: 22018: 32019: 42020: 42021: 32022: 22023: 22024: 62025: 142026: 9
2007peak 14 in 20252026

Frequent co-authors

Papers

showing 1–25 of 55 RSS

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Yujun Lin, Song Han, Huizi Mao +2 more

cs.CV · · cited 1,742× · arXiv ↗ · more like this ·

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet…

HAQ: Hardware-Aware Automated Quantization with Mixed Precision

Kuan Wang, Zhijian Liu, Yujun Lin +2 more

cs.CV · · cited 1,166× · arXiv ↗ · more like this ·

Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such…

Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution

Haotian Tang, Zhijian Liu, Shengyu Zhao +4 more

cs.CV · · cited 892× · arXiv ↗ · more like this ·

Self-driving cars need to understand 3D scenes efficiently and accurately in order to drive safely. Given the limited hardware resources, existing 3D perception models are not able to recognize small instances (e.g., pedestrians, cyclists) very well due to the low-resolution voxelization and aggressive downsampling. To this end, we propose Sparse Point-Voxel Convolution (SPVConv), a lightweight 3D module that equips the vanilla Sparse Convolution with the high-resolution point-based branch. With negligible overhead, this point-based branch is able to preserve the fine details even from large outdoor scenes. To explore the spectrum of efficient 3D models, we first define a flexible architecture design space based on SPVConv, and we then present 3D Neural Architecture Search (3D-NAS) to search the optimal network architecture over this diverse design space efficiently and effectively.…

Point-Voxel CNN for Efficient 3D Deep Learning

Zhijian Liu, Haotian Tang, Yujun Lin +1 more

cs.CV · · cited 870× · arXiv ↗ · more like this ·

We present Point-Voxel CNN (PVCNN) for efficient, fast 3D deep learning. Previous work processes 3D data using either voxel-based or point-based NN models. However, both approaches are computationally inefficient. The computation cost and memory footprints of the voxel-based models grow cubically with the input resolution, making it memory-prohibitive to scale up the resolution. As for point-based networks, up to 80% of the time is wasted on structuring the sparse data which have rather poor memory locality, not on the actual feature extraction. In this paper, we propose PVCNN that represents the 3D input data in points to reduce the memory consumption, while performing the convolutions in voxels to reduce the irregular, sparse data access and improve the locality. Our PVCNN model is both memory and computation efficient. Evaluated on semantic and part segmentation datasets, it achieves…

MCUNet: Tiny Deep Learning on IoT Devices

Ji Lin, Wei-Ming Chen, Yujun Lin +3 more

cs.CV · · cited 819× · arXiv ↗ · more like this ·

Machine learning on tiny IoT devices based on microcontroller units (MCU) is appealing but challenging: the memory of microcontrollers is 2-3 orders of magnitude smaller even than mobile phones. We propose MCUNet, a framework that jointly designs the efficient neural architecture (TinyNAS) and the lightweight inference engine (TinyEngine), enabling ImageNet-scale inference on microcontrollers. TinyNAS adopts a two-stage neural architecture search approach that first optimizes the search space to fit the resource constraints, then specializes the network architecture in the optimized search space. TinyNAS can automatically handle diverse constraints (i.e.device, latency, energy, memory) under low search costs.TinyNAS is co-designed with TinyEngine, a memory-efficient inference library to expand the search space and fit a larger model. TinyEngine adapts the memory scheduling according to…

Lite Transformer with Long-Short Range Attention

Zhanghao Wu, Zhijian Liu, Ji Lin +2 more

cs.CL · · cited 404× · arXiv ↗ · more like this ·

Transformer has become ubiquitous in natural language processing (e.g., machine translation, question answering); however, it requires enormous amount of computations to achieve high performance, which makes it not suitable for mobile applications that are tightly constrained by the hardware resources and battery. In this paper, we present an efficient mobile NLP architecture, Lite Transformer to facilitate deploying mobile NLP applications on edge devices. The key primitive is the Long-Short Range Attention (LSRA), where one group of heads specializes in the local context modeling (by convolution) while another group specializes in the long-distance relationship modeling (by attention). Such specialization brings consistent improvement over the vanilla transformer on three well-established language tasks: machine translation, abstractive summarization, and language modeling. Under…

QuantumNAS: Noise-Adaptive Search for Robust Quantum Circuits

Hanrui Wang, Yongshan Ding, Jiaqi Gu +5 more

quant-ph · · cited 310× · arXiv ↗ · more like this ·

Quantum noise is the key challenge in Noisy Intermediate-Scale Quantum (NISQ) computers. Previous work for mitigating noise has primarily focused on gate-level or pulse-level noise-adaptive compilation. However, limited research efforts have explored a higher level of optimization by making the quantum circuits themselves resilient to noise. We propose QuantumNAS, a comprehensive framework for noise-adaptive co-search of the variational circuit and qubit mapping. Variational quantum circuits are a promising approach for constructing QML and quantum simulation. However, finding the best variational circuit and its optimal parameters is challenging due to the large design space and parameter training cost. We propose to decouple the circuit search and parameter training by introducing a novel SuperCircuit. The SuperCircuit is constructed with multiple layers of pre-defined parameterized…

Big Data Driven Mobile Traffic Understanding and Forecasting: A Time Series Approach

Fengli Xu, Yujun Lin, Jiaxin Huang +4 more

Transportation · IEEE Transactions on Services Computing · · cited 248× · DOI ↗ · more like this ·

Understanding and forecasting mobile traffic of large scale cellular networks is extremely valuable for service providers to control and manage the explosive mobile data, such as network planning, load balancing, and data pricing mechanisms. This paper targets at extracting and modeling traffic patterns of 9,000 cellular towers deployed in a metropolitan city. To achieve this goal, we design, implement, and evaluate a time series analysis approach that is able to decompose large scale mobile traffic into regularity and randomness components. Then, we use time series prediction to forecast the traffic patterns based on the regularity components. Our study verifies the effectiveness of our utilized time series decomposition method, and shows the geographical distribution of the regularity and randomness component. Moreover, we reveal that high predictability of the regularity component…

Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Han Cai, Ji Lin, Yujun Lin +5 more

cs.LG · · cited 154× · arXiv ↗ · more like this ·

Deep neural networks (DNNs) have achieved unprecedented success in the field of artificial intelligence (AI), including computer vision, natural language processing and speech recognition. However, their superior performance comes at the considerable cost of computational complexity, which greatly hinders their applications in many resource-constrained devices, such as mobile phones and Internet of Things (IoT) devices. Therefore, methods and techniques that are able to lift the efficiency bottleneck while preserving the high accuracy of DNNs are in great demand in order to enable numerous edge AI applications. This paper provides an overview of efficient deep learning methods, systems and applications. We start from introducing popular model compression methods, including pruning, factorization, quantization as well as compact model design. To reduce the large design cost of these…

TorchSparse: Efficient Point Cloud Inference Engine

Haotian Tang, Zhijian Liu, Xiuyu Li +2 more

cs.LG · · cited 148× · arXiv ↗ · more like this ·

Deep learning on point clouds has received increased attention thanks to its wide applications in AR/VR and autonomous driving. These applications require low latency and high accuracy to provide real-time user experience and ensure user safety. Unlike conventional dense workloads, the sparse and irregular nature of point clouds poses severe challenges to running sparse CNNs efficiently on the general-purpose hardware. Furthermore, existing sparse acceleration techniques for 2D images do not translate to 3D point clouds. In this paper, we introduce TorchSparse, a high-performance point cloud inference engine that accelerates the sparse convolution computation on GPUs. TorchSparse directly optimizes the two bottlenecks of sparse convolution: irregular computation and data movement. It applies adaptive matrix multiplication grouping to trade computation for better regularity, achieving…

NAAS: Neural Accelerator Architecture Search

Yujun Lin, Mengtian Yang, Song Han

cs.LG · · cited 82× · arXiv ↗ · more like this ·

Data-driven, automatic design space exploration of neural accelerator architecture is desirable for specialization and productivity. Previous frameworks focus on sizing the numerical architectural hyper-parameters while neglect searching the PE connectivities and compiler mappings. To tackle this challenge, we propose Neural Accelerator Architecture Search (NAAS) which holistically searches the neural network architecture, accelerator architecture, and compiler mapping in one optimization loop. NAAS composes highly matched architectures together with efficient mapping. As a data-driven approach, NAAS rivals the human design Eyeriss by 4.4x EDP reduction with 2.7% accuracy improvement on ImageNet under the same computation resource, and offers 1.4x to 3.5x EDP reduction than only sizing the architectural hyper-parameters.

Long live TIME

Yi Cai, Yujun Lin, Lixue Xia +4 more

Computer Vision and Pattern Recognition · · cited 36× · DOI ↗ · more like this ·

Deeper and larger Neural Networks (NNs) have made breakthroughs in many fields. While conventional CMOS-based computing platforms are hard to achieve higher energy efficiency. RRAM-based systems provide a promising solution to build efficient Training-In-Memory Engines (TIME). While the endurance of RRAM cells is limited, it's a severe issue as the weights of NN always need to be updated for thousands to millions of times during training. Gradient sparsification can address this problem by dropping off most of the smaller gradients but introduce unacceptable computation cost. We proposed an effective framework, SGS-ARS, including Structured Gradient Sparsification (SGS) and Aging-aware Row Swapping (ARS) scheme, to guarantee write balance across whole RRAM crossbars and prolong the lifetime of TIME. Our experiments demonstrate that 356× lifetime extension is achieved when TIME is…

AutoML for Architecting Efficient and Specialized Neural Networks

Han Cai, Ji Lin, Yujun Lin +5 more

Computer Vision and Pattern Recognition · IEEE Micro · · cited 26× · DOI ↗ · more like this ·

Efficient deep learning inference requires algorithm and hardware codesign to enable specialization: we usually need to change the algorithm to reduce memory footprint and improve energy efficiency. However, the extra degree of freedom from the neural architecture design makes the design space much larger: it is not only about designing the hardware architecture but also codesigning the neural architecture to fit the hardware architecture. It is difficult for human engineers to exhaust the design space by heuristics. We propose design automation techniques for architecting efficient neural networks given a target hardware platform. We investigate automatically designing specialized and fast models, auto channel pruning, and auto mixed-precision quantization. We demonstrate that such learning-based, automated design achieves superior performance and efficiency than the rule-based human…

Design Automation for Efficient Deep Learning Computing

Song Han, Han Cai, Ligeng Zhu +4 more

cs.LG · · cited 20× · arXiv ↗ · more like this ·

Efficient deep learning computing requires algorithm and hardware co-design to enable specialization: we usually need to change the algorithm to reduce memory footprint and improve energy efficiency. However, the extra degree of freedom from the algorithm makes the design space much larger: it's not only about designing the hardware but also about how to tweak the algorithm to best fit the hardware. Human engineers can hardly exhaust the design space by heuristics. It's labor consuming and sub-optimal. We propose design automation techniques for efficient neural networks. We investigate automatically designing specialized fast models, auto channel pruning, and auto mixed-precision quantization. We demonstrate such learning-based, automated design achieves superior performance and efficiency than rule-based human design. Moreover, we shorten the design cycle by 200x than previous work…

An improved clustering method based on k-means

Yujun Lin, Ting Luo, Sheng Yao +3 more

Artificial Intelligence · · cited 20× · DOI ↗ · more like this ·

In this paper, an improved clustering method based on k-means is proposed. The proposed method consists of two major stages split and merge stages. Initially k-means method is employed in the dataset, and in the split stage, each cluster will be split into smaller clusters with k-mean repeatedly if they are sparse. Furthermore, in the merge stage, the average distance is employed for merging standard. Experiments are tested on real and synthetic datasets. Experimental results demonstrate the proposed clustering method can detect clusters with different sizes, shapes and densities. Moreover, it outperforms the traditional k-means and single-link clustering method.

On the Understanding of Interdependency of Mobile App Usage

Jiaxin Huang, Fengli Xu, Yujun Lin +1 more

Computer Vision and Pattern Recognition · · cited 13× · DOI ↗ · more like this ·

With the rising popularity of smartphones and the rapid growth of mobile applications, understanding the app usage behavior of mobile users is of growing importance for both app designers and service providers. Different from previous studies mining the correlation between apps and physical world factors, e.g. location, time, etc., in this paper we focus on the interdependency among apps and try to address a series of research problems: what apps are frequently used together? What are the relations between apps (strengthen or undermine the use of each other) belonged to each category? To answer these questions, we employ the frequent pattern mining algorithm to a large-scale real-world dataset, which includes more than 1.7 million users and 5 billion app usage logs, and find out frequent app-sets and association rules with interesting insights. These results are usefulness in app…

A Fine-Grained Sparse Accelerator for Multi-Precision DNN

Shulin Zeng, Yujun Lin, Shuang Liang +6 more

Artificial Intelligence · · cited 11× · DOI ↗ · more like this ·

Neural Networks (NNs) have made a significant breakthrough in many fields, while they also pose a great challenge to hardware platforms since the state-of-the-art neural networks are both communicational- and computational-intensive. Researchers proposed model compression algorithms using sparsification and quantization, along with specific hardware architecture designs, to accelerate various applications. However, the irregularity of memory access caused by the sparsity severely damages the regularity of intensive computation loops. Therefore, the architecture design for sparse neural networks is crucial to better software and hardware co-design for neural network applications. To face these challenges, this paper first analyzes the computation patterns of different NN structures and unify them into the form of sparse matrix-vector multiplication, sparse matrix-matrix multiplication…

SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

Enze Xie, Junsong Chen, Junyu Chen +8 more

cs.CV · · cited 4× · arXiv ↗ · more like this ·

We introduce Sana, a text-to-image framework that can efficiently generate images up to 4096$\times$4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU. Core designs include: (1) Deep compression autoencoder: unlike traditional AEs, which compress images only 8$\times$, we trained an AE that can compress images 32$\times$, effectively reducing the number of latent tokens. (2) Linear DiT: we replace all vanilla attention in DiT with linear attention, which is more efficient at high resolutions without sacrificing quality. (3) Decoder-only text encoder: we replaced T5 with modern decoder-only small LLM as the text encoder and designed complex human instruction with in-context learning to enhance the image-text alignment. (4) Efficient training and sampling: we propose…

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

Xingyang Li, Dongyun Zou, Shining Zhang +8 more

cs.CV · · cited 3× · arXiv ↗ · more like this ·

Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values. Prior work smooths queries and keys, but value outliers follow no fixed channel or spatiotemporal structure and remain the dominant source of output error. Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. We propose VC-Attention, a training-free low-bit attention framework that addresses both by pairing Value smoothing with a fused probability Cast. V-Smooth reorders…

“I Am Perceiving and Overcoming It”: A Qualitative Study on How Women View Gendered Peer Pressure in Public Relations

Yujun Lin, Yang Yu, Yirui Jiang

Gender Studies · Journal of Public Relations Research · · cited 2× · DOI ↗ · more like this ·

Feminist scholarship in public relations (PR) has grown significantly, particularly in addressing gender stereotypes and inequitable treatment. However, gendered peer pressure, a prevalent yet underexplored phenomenon, remains understudied, especially regarding its contributing factors and strategies for overcoming it from a theoretical perspective. This qualitative study explores the experiences of nineteen female PR professionals through in-depth interviews, identifying key factors contributing to gendered peer pressure and the strategies women employ to navigate it. The findings, analyzed from a feminist perspective, introduce a new theoretical concept, performativity dynamics, which explains how women negotiate and resolve gendered peer pressure within the field. By advancing feminist PR scholarship, this study provides a critical framework for understanding gendered dynamics in the…

Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter

Qinghao Hu, Shang Yang, Junxian Guo +7 more

cs.LG · · cited 1× · arXiv ↗ · more like this ·

The emergence of Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving. However, training these reasoning models, typically using Reinforcement Learning (RL), encounters critical efficiency bottlenecks: response generation during RL training exhibits a persistent long-tail distribution, where a few very long responses dominate execution time, wasting resources and inflating costs. To address this, we propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding. Applying speculative decoding in RL is challenging due to the dynamic workloads, evolving target model, and draft model training overhead. TLT overcomes these obstacles with two synergistic components: (1) Adaptive Drafter, a lightweight draft model trained continuously on idle GPUs…

LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention

Shang Yang, Junxian Guo, Haotian Tang +7 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

Large language models (LLMs) have shown remarkable potential in processing long sequences and complex reasoning tasks, yet efficiently serving these models remains challenging due to the quadratic computational complexity of attention in the prefilling stage and the large memory footprint of the KV cache in the decoding stage. To address these issues, we introduce LServe, an efficient system that accelerates long-sequence LLM serving via hybrid sparse attention. This method unifies different hardware-friendly, structured sparsity patterns for both prefilling and decoding attention into a single framework, where computations on less important tokens are skipped block-wise. LServe demonstrates the compatibility of static and dynamic sparsity in long-context LLM attention. This design enables multiplicative speedups by combining these optimizations. Specifically, we convert half of the…

SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer

Enze Xie, Junsong Chen, Yuyang Zhao +11 more

cs.CV · · cited 1× · arXiv ↗ · more like this ·

This paper presents SANA-1.5, a linear Diffusion Transformer for efficient scaling in text-to-image generation. Building upon SANA-1.0, we introduce three key innovations: (1) Efficient Training Scaling: A depth-growth paradigm that enables scaling from 1.6B to 4.8B parameters with significantly reduced computational resources, combined with a memory-efficient 8-bit optimizer. (2) Model Depth Pruning: A block importance analysis technique for efficient model compression to arbitrary sizes with minimal quality loss. (3) Inference-time Scaling: A repeated sampling strategy that trades computation for model capacity, enabling smaller models to match larger model quality at inference time. Through these strategies, SANA-1.5 achieves a text-image alignment score of 0.81 on GenEval, which can be further improved to 0.96 through inference scaling with VILA-Judge, establishing a new SoTA on…

Data-Driven Robust Dynamic Wireless Charging Lane Planning with Tunable Profit--Risk Trade-off and Modified User Equilibrium

Xin Cui, Yujun Lin, Ziang Liu +2 more

eess.SY · · cited 0× · arXiv ↗ · more like this ·

Dynamic wireless charging (DWC) has attracted increasing attention as a complement to conventional plug-in charging for electric vehicles (EVs). However, existing planning methods for dynamic wireless charging lanes (DWCLs) overlook realistic EV routing and charging behavior and lack explicit out-of-sample risk control under traffic-demand and renewable-energy uncertainties. To address these limitations, this paper proposes a data-driven two-stage robust planning framework that coordinates DWCL deployment with the operation of coupled traffic--power networks (CTPNs). A data-driven uncertainty set is constructed from joint historical scenarios to preserve spatio-temporal correlations without imposing parametric distributional assumptions. A modified user-equilibrium (MUE) model is proposed to capture realistic EV routing and charging behavior by incorporating state-of-charge (SoC)…

Channel Knowledge Map Enabled Low-Complexity Dynamic Radio Environment Reconstruction

Yujun Lin, Zhiqiang Xiao, Hao Wu +3 more

eess.SP · · cited 0× · arXiv ↗ · more like this ·

Accurate and timely radio environment reconstruction is important but challenging under particularly dynamic transmitter configurations. The conventional methods such as compressed sensing (CS), Kriging method or U-Net typically require environment measurements and reconstruction overhead for radio environment updating as the transmitter locations or radiation patterns change. In this paper, we propose a novel channel knowledge map (CKM)-enabled dynamic radio environment reconstruction method for efficient radio map updating. Specifically, the recently proposed CKM can store reusable path-level propagation knowledge that is decoupled from the transmitter-side radiation characteristics. We can leverage CKM for lightweight forward radio map generation as the transmitter locations and radiation patterns are known, without requiring new target-map measurements. Simulation results show that…