Author

Jeff Pool

20 papers · 1966–2026

1967peak 3 in 20172026

Frequent co-authors

Papers

showing 1–20 of 20 RSS

Learning both Weights and Connections for Efficient Neural Networks

Song Han, Jeff Pool, John Tran +1 more

cs.NE · June 2015 · cited 8,047× · arXiv ↗ · more like this ·

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9x, from 61 million to 6.7 million, without incurring accuracy…

Accelerating Sparse Deep Neural Networks

Asit Mishra, Jorge Albericio Latorre, Jeff Pool +5 more

cs.LG · 16 Apr 2021 · cited 364× · arXiv ↗ · more like this ·

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4…

Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

Huizi Mao, Song Han, Jeff Pool +4 more

cs.LG · May 2017 · cited 259× · arXiv ↗ · more like this ·

Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as well as the prediction accuracy. Coarse-grained pruning creates regular sparsity patterns, making it more amenable for hardware acceleration but more challenging to maintain the same accuracy. In this paper we quantitatively measure the trade-off between sparsity regularity and prediction accuracy, providing insights in how to maintain accuracy while having more a more structured sparsity pattern. Our experimental results show that coarse-grained pruning can achieve a sparsity ratio similar to unstructured pruning without loss of accuracy. Moreover, due to the index saving…

Exploring the Granularity of Sparsity in Convolutional Neural Networks

Huizi Mao, Song Han, Jeff Pool +4 more

Computer Vision and Pattern Recognition · July 2017 · cited 198× · DOI ↗ · more like this ·

Sparsity helps reducing the computation complexity of DNNs by skipping the multiplication with zeros. The granularity of sparsity affects the efficiency of hardware architecture and the prediction accuracy. In this paper we quantitatively measure the accuracy-sparsity relationship with different granularity. Coarse-grained sparsity brings more regular sparsity pattern, making it easier for hardware acceleration, and our experimental results show that coarsegrained sparsity have very small impact on the sparsity ratio given no loss of accuracy. Moreover, due to the index saving effect, coarse-grained sparsity is able to obtain similar or even better compression rates than fine-grained sparsity at the same accuracy threshold. Our analysis, which is based on the framework of a recent sparse convolutional neural network (SCNN) accelerator, further demonstrates that it saves 30% – 35% of…

Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks

Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee +2 more

cs.LG · 3 May 2017 · cited 196× · arXiv ↗ · more like this ·

Popular deep learning frameworks require users to fine-tune their memory usage so that the training data of a deep neural network (DNN) fits within the GPU physical memory. Prior work tries to address this restriction by virtualizing the memory usage of DNNs, enabling both CPU and GPU memory to be utilized for memory allocations. Despite its merits, virtualizing memory can incur significant performance overheads when the time needed to copy data back and forth from CPU memory is higher than the latency to perform the computations required for DNN forward and backward propagation. We introduce a high-performance virtualization strategy based on a "compressing DMA engine" (cDMA) that drastically reduces the size of the data structures that are targeted for CPU-side allocations. The cDMA engine offers an average 2.6x (maximum 13.8x) compression ratio by exploiting the sparsity inherent in…

Efficient Sparse-Winograd Convolutional Neural Networks

Xingyu Liu, Jeff Pool, Song Han +1 more

cs.CV · 18 Feb 2018 · cited 127× · arXiv ↗ · more like this ·

Convolutional Neural Networks (CNNs) are computationally intensive, which limits their application on mobile devices. Their energy is dominated by the number of multiplies needed to perform the convolutions. Winograd's minimal filtering algorithm (Lavin, 2015) and network pruning (Han et al., 2015) can reduce the operation count, but these two methods cannot be directly combined $-$ applying the Winograd transform fills in the sparsity in both the weights and the activations. We propose two modifications to Winograd-based CNNs to enable these methods to exploit sparsity. First, we move the ReLU operation into the Winograd domain to increase the sparsity of the transformed activations. Second, we prune the weights in the Winograd domain to exploit static weight sparsity. For models on CIFAR-10, CIFAR-100 and ImageNet datasets, our method reduces the number of multiplications by…

DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Song Han, Jeff Pool, Sharan Narang +9 more

cs.CV · July 2016 · cited 84× · arXiv ↗ · more like this ·

Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1…

Energy-precision tradeoffs in mobile Graphics Processing Units

Jeff Pool, Anselmo Lastra, Montek Singh

Hardware and Architecture · October 2008 · cited 31× · DOI ↗ · more like this ·

In mobile devices, limiting the Graphics Processing Unitpsilas (GPUpsilas) energy usage is of great importance to extending battery life. This paper focuses on the first stage of the graphics processor pipeline - the vertex transformation stage - and introduces an approach to lowering its switching activity by reducing the precision of arithmetic operations. As a result, the approach enables a tradeoff between energy efficiency and the quality of the rendered image. This paper makes the following specific contributions: 1) a transition-based energy model for quantifying energy consumed as a function of arithmetic precision, and 2) detailed simulation results on several real-world graphics applications to evaluate the tradeoff between energy and precision. In most examples, over 23% of the energy can be saved by lowering arithmetic precision while still maintaining a faithful…

An energy model for graphics processing units

Jeff Pool, Anselmo Lastra, Montek Singh

Hardware and Architecture · October 2010 · cited 27× · DOI ↗ · more like this ·

We present an energy model for a graphics processing unit (GPU) that is based on the amount and type of work performed in various parts of the unit. By designing and running directed tests on a GPU, we measure the energy consumed when performing different arithmetic and memory operations, allowing us to accurately predict the energy that any arbitrary mix of operations will take. With some knowledge of how data travels through and is transformed by the graphics pipeline, we can predict how many of each operation will occur for a given scene, leading to an estimate of the energy usage. We validate our model against different types of existing graphical applications. With an average difference of 3% from measured energy under typical workloads, our model can be used for various purposes. In this work, we explore and present two use cases: 1) predicting the energy performance of…

Precision selection for energy-efficient pixel shaders

Jeff Pool, Anselmo Lastra, Montek Singh

Hardware and Architecture · 5 Aug 2011 · cited 21× · DOI ↗ · more like this ·

In this work, we seek to realize energy savings in modern pixel shaders by reducing the precision of their arithmetic. We explore three schemes for controlling this reduction. The first is a static analysis technique, which analyzes shader programs to choose precision with guaranteed error bounds. This approach may be too conservative in practice since it cannot take advantage of run-time information, so we also examine two methods that take the actual data values into account - a programmer-directed approach and a closed-loop error-tracking approach, both of which can lead to higher savings. To use this last method, we developed several heuristics to control how the precisions will change over time. We simulate several series of frames from commercial applications to evaluate the performance of these different schemes. The average savings found by the static and dynamic approaches are…

Energy-precision tradeoffs in the graphics pipeline

Anselmo Lastra, Montek Singh, Jeff Pool

Hardware and Architecture · 2012 · cited 12× · OpenAlex ↗ · more like this ·

The energy consumption of a graphics processing unit (GPU) is an important factor in its design, whether for a server, desktop, or mobile device. Mobile products, such as smart phones, tablets, and laptop computers, rely on batteries to function; the less the demand for power is on these batteries, the longer they will last before needing to be recharged. GPUs used in servers and desktops, while not dependent on a battery for operation, are still limited by the efficiency of power supplies and heat dissipation techniques. In this dissertation, I propose to lower the energy consumption of GPUs by reducing the precision of floating-point arithmetic in the graphics pipeline and the data sent and stored on- and off-chip. The key idea behind this work is twofold: energy can be saved through a systematic and targeted reduction in the number of bits 1) computed and 2) communicated. Reducing…

Lossless compression of variable-precision floating-point buffers on GPUs

Jeff Pool, Anselmo Lastra, Montek Singh

Computational Theory and Mathematics · 9 Mar 2012 · cited 9× · DOI ↗ · more like this ·

In this work, we explore the lossless compression of 32-bit floating-point buffers on graphics hardware. We first adapt a state-of-the-art 16-bit floating-point color and depth buffer compression scheme for operation on 32-bit data and propose two specific enhancements: dynamic bucket selection and a Fibonacci encoder. Next, we describe a unified codec for any type of floating-point buffer: color, depth, geometry, and GPGPU data. We also propose a method to further compress variable-precision data. Finally, we test our techniques on color, depth, and geometry buffers from existing applications. Using our enhancements to an existing technique, we have improved bandwidth savings by an average of 1.26x. Our unified codec achieved average bandwidth savings of 1.5x, 7.9x, and 2.9x for color (including buffers incompressible by past work), depth, and geometry buffers. Even higher savings were…

Power-Gated Arithmetic Circuits for Energy-Precision Tradeoffs in Mobile Graphics Processing Units

Jeff Pool, Anselmo Lastra, Montek Singh

Hardware and Architecture · Journal of Low Power Electronics · April 2011 · cited 9× · DOI ↗ · more like this ·

In mobile devices, limiting the Graphics Processing Unit's (GPU's) energy usage is of great importance to extending battery life. This work shows that significant energy savings can be obtained by reducing the precision of graphics computations, yet maintaining acceptable quality of the final rendered image. In particular, we focus on a portion of a typical graphics processor pipeline— the vertex transformation stage—and evaluate the tradeoff between energy efficiency and image fidelity. We first develop circuit-level designs of arithmetic components whose precision can be varied dynamically with fine-grained power gating techniques. Spice simulation is used to characterize each component's energy consumption, based on which a system-level energy model for the entire vertex stage is developed. We then use this energy model in conjunction with a graphics hardware simulator to determine…

Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training

Maohua Zhu, Jason Clemons, Jeff Pool +3 more

cs.LG · June 2018 · cited 7× · arXiv ↗ · more like this ·

Exploiting sparsity enables hardware systems to run neural networks faster and more energy-efficiently. However, most prior sparsity-centric optimization techniques only accelerate the forward pass of neural networks and usually require an even longer training process with iterative pruning and retraining. We observe that artificially inducing sparsity in the gradients of the gates in an LSTM cell has little impact on the training quality. Further, we can enforce structured sparsity in the gate gradients to make the LSTM backward pass up to 45% faster than the state-of-the-art dense approach and 168% faster than the state-of-the-art sparsifying method on modern GPUs. Though the structured sparsifying method can impact the accuracy of a model, this performance gap can be eliminated by mixing our sparse training method and the standard dense training method. Experimental results show that…

Self-Supervised GAN Compression

Chong Yu, Jeff Pool

cs.LG · 12 Jul 2020 · cited 4× · arXiv ↗ · more like this ·

Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters. These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied to image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of…

MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models

Gongfan Fang, Hongxu Yin, Saurav Muralidharan +5 more

cs.AI · September 2024 · cited 1× · arXiv ↗ · more like this ·

Large Language Models (LLMs) are distinguished by their massive parameter counts, which typically result in significant redundancy. This work introduces MaskLLM, a learnable pruning method that establishes Semi-structured (or ``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during inference. Instead of developing a new importance criterion, MaskLLM explicitly models N:M patterns as a learnable distribution through Gumbel Softmax sampling. This approach facilitates end-to-end training on large-scale datasets and offers two notable advantages: 1) High-quality Masks - our method effectively scales to large datasets and learns accurate masks; 2) Transferability - the probabilistic modeling of mask distribution enables the transfer learning of sparsity across domains or tasks. We assessed MaskLLM using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3…

Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs

Esha Choukse, Michael Sullivan, Mike O'Connor +4 more

cs.AR · March 2019 · cited 1× · arXiv ↗ · more like this ·

GPUs offer orders-of-magnitude higher memory bandwidth than traditional CPU-only systems. However, GPU device memory tends to be relatively small and the memory capacity can not be increased by the user. This paper describes Buddy Compression, a scheme to increase both the effective GPU memory capacity and bandwidth while avoiding the downsides of conventional memory-expanding strategies. Buddy Compression compresses GPU memory, splitting each compressed memory entry between high-speed device memory and a slower-but-larger disaggregated memory pool (or system memory). Highly-compressible memory entries can thus be accessed completely from device memory, while incompressible entries source their data using both on and off-device accesses. Increasing the effective GPU memory capacity enables us to run larger-memory-footprint HPC workloads and larger batch-sizes or models for DL workloads…

Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip

Feiwen Zhu, Jeff Pool, Michael Andersch +2 more

cs.NE · 26 Apr 2018 · cited 1× · arXiv ↗ · more like this ·

Recurrent Neural Networks (RNNs) are powerful tools for solving sequence-based problems, but their efficacy and execution time are dependent on the size of the network. Following recent work in simplifying these networks with model pruning and a novel mapping of work onto GPUs, we design an efficient implementation for sparse RNNs. We investigate several optimizations and tradeoffs: Lamport timestamps, wide memory loads, and a bank-aware weight layout. With these optimizations, we achieve speedups of over 6x over the next best algorithm for a hidden layer of size 2304, batch size of 4, and a density of 30%. Further, our technique allows for models of over 5x the size to fit on a GPU for a speedup of 2x, enabling larger networks to help advance the state-of-the-art. We perform case studies on NMT and speech recognition tasks in the appendix, accelerating their recurrent layers by up to…

Use of Microwave in Automated Pipeline System Reduces Costs, Increases Efficiency

I.D. Messersmith, Jeff Pool

Information Systems · Journal of Petroleum Technology · July 1966 · cited 0× · DOI ↗ · more like this ·

Abstract Use of microwave as the communications link in the automation and central control of a pipeline has proven to be an outstanding success. This article describes the centralized dispatching group that maintains control and supervision over the entire pipeline system; the design of pump stations and equipment involved; and the supervisory maintenance program used in training company personnel. Introduction The dispatching and control of Magnolia Pipe Line Co.'s crude and LPG pipelines from Midland, Tex., to Mobil Oil Co.'s refinery in Beaumont, Tex. are under the direct control and complete supervision of a centralized dispatching group headquartered in Dallas (Fig. 1). In all, 11 stations are under remote supervision and control over a company-owned microwave system serving as the communications link. Eight stations are controlled by solid-state, continuous scan supervisory…

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

Mikail Khona, Aditya Vavre, Boxiang Wang +11 more

cs.LG · 13 Jul 2026 · arXiv ↗ · more like this ·

Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale. In this work, we adapt and enhance preconditioned gradient methods to overcome the practical challenges of large-scale LLM pretraining. We first identify instabilities in SOAP at large batch sizes and propose algorithmic modifications including per-step QR orthogonalization and improved preconditioning strategies that eliminate loss spikes and enable stable training in these regimes. We then present a unified empirical study of SOAP, Muon, and AdamW using update-RMS matching to ensure fair learning rate transfer across optimizers. As part of this analysis, we empirically evaluate the orthogonalization quality of Muon. Our experiments on multi-billion-parameter models trained on trillions of tokens reveal that…