arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Sean Lie

18 papers · 2003–2025

2003: 12004: 12005: 02006: 12007: 02008: 12009: 02010: 02011: 12012: 02013: 22014: 02015: 02016: 02017: 02018: 02019: 02020: 02021: 02022: 02023: 32024: 42025: 4
2003peak 4 in 20242025

Frequent co-authors

Papers

showing 1–18 of 18 RSS

Cerebras Architecture Deep Dive: First Look Inside the Hardware/Software Co-Design for Deep Learning

Sean Lie

Hardware and Architecture · IEEE Micro · · cited 57× · DOI ↗ · more like this ·

The compute and memory demands for deep learning and machine learning (ML) have increased by several orders of magnitude in just the last couple of years, and there is no end in sight. Traditional improvements in processor performance alone struggle to keep up with the exponential demand. A new chip architecture co-designed with the ML algorithms can be better equipped to satisfy this unprecedented demand and enable the ML workloads of the future. This article describes the Cerebras architecture and how it is designed specifically with this purpose, from the ground up, as a wafer-sized chip to enable emerging extreme-scale ML models. It uses fine-grained data flow compute cores to accelerate unstructured sparsity, distributed static random-access memory for full memory bandwidth to the data paths, and a specially designed on-chip and off-chip interconnect for ML training. With these…

Inside the Cerebras Wafer-Scale Cluster

Sean Lie

Urban Studies · IEEE Micro · · cited 12× · DOI ↗ · more like this ·

The compute and memory demands of Machine Learning (ML) have driven the industry to use clusters of thousands of GPUs to train state-of-the-art models. However, scaling performance on GPU clusters is inherently complex, requires many forms of parallelism, and often has poor scaling performance. The Cerebras Wafer-Scale Cluster provides ExaFLOP level compute while avoiding these limitations by using only data parallelism. The cluster is built around the CS-2 system with the Wafer-Scale Engine-2 (WSE-2) processor that enables even the largest models to run on a single chip without partitioning. This unique property allows the cluster architecture to decouple memory and compute. Model weights are stored in an external memory device, MemoryX, and compute scaling is enabled using a special broadcast-reduce fabric, SwarmX. This article describes the challenges of GPU scaling and details the…

SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models

Vithursan Thangarasa, Abhay Gupta, William Marshall +5 more

cs.LG · · cited 7× · arXiv ↗ · more like this ·

The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural Language Processing (NLP). Instead of directly training on a downstream task, language models are first pre-trained on large datasets with cross-domain knowledge (e.g., Pile, MassiveText, etc.) and then fine-tuned on task-specific data (e.g., natural language generation, text summarization, etc.). Scaling the model and dataset size has helped improve the performance of LLMs, but unfortunately, this also lead to highly prohibitive computational costs. Pre-training LLMs often require orders of magnitude more FLOPs than fine-tuning and the model capacity often remains the same between the two phases. To achieve training efficiency w.r.t training FLOPs, we propose to decouple the model capacity between the two phases and introduce Sparse Pre-training and Dense Fine-tuning (SPDF). In this work…

A novel system architecture for web scale applications using lightweight CPUs and virtualized I/O

Kshitij Sudan, S. Balakrishnan, Sean Lie +4 more

Computer Networks and Communications · · cited 6× · DOI ↗ · more like this ·

Large web-scale applications typically use a distributed platform, like clusters of commodity servers, to achieve scalable and low-cost processing. The Map-Reduce framework and its open-source implementation, Hadoop, is commonly used to program these applications. Since these applications scale well with an increased number of servers, the cluster size is an important parameter. Cluster size however is constrained by power consumption. In this paper we present a system that uses low-power CPUs to increase the cluster size in a fixed power budget. Using low-power CPUs leads to the situation where the majority of a server's power is now consumed by the I/O sub-system. To overcome this, we develop a virtualized I/O sub-system where multiple servers share I/O resources. An ASIC based high-bandwidth interconnect fabric, and FPGA based I/O cards implement this virtualized I/O. The resulting…

Hardware Transactional Memory

Sean Lie, Krste Asanović, Bradley C. Kuszmaul +1 more

Computer Networks and Communications · DSpace@MIT (Massachusetts Institute of Technology) · · cited 2× · OpenAlex ↗ · more like this ·

This work shows how hardware transactional memory (HTM) can be implemented to support transactions of arbitrarily large size, while ensuring that small transactions run efficiently. Our implementation handles small transactions similar to Herlihy and Moss's scheme in that it holds tentative updates in a cache. Unlike their scheme, which uses a special fully associative cache, ours augments the ordinary processor cache and provides a mechanism to handle cache spills of uncommitted transactional data. Consequently, our scheme runs faster for small transactions while correctly handling transactions of arbitrarily large size. <br/>Although transactions are small in the common case, we argue that HTM should not restrict the size of transactions, because it complicates the programmer/compiler model and precludes some important programs from exploiting transactional memory. We show that the…

MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models

Mugilan Ganesan, Shane Segal, Ankur Aggarwal +3 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Speculative decoding significantly accelerates language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. However, applying this technique to vision-language models (VLMs) presents two fundamental challenges: small language models that could serve as efficient drafters lack the architectural components to process visual inputs, and their token predictions fail to match those of VLM target models that consider visual context. We introduce Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (MASSV), which transforms existing small language models into effective multimodal drafters through a two-phase approach. MASSV first connects the target VLM's vision encoder to the draft model via a lightweight trainable projector, then applies self-distilled visual…

SD$^2$: Self-Distilled Sparse Drafters

Mike Lasby, Nish Sinnadurai, Valavan Manohararajah +3 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Speculative decoding is a powerful technique for reducing the latency of Large Language Models (LLMs), offering a fault-tolerant framework that enables the use of highly compressed draft models. In this work, we introduce Self-Distilled Sparse Drafters (SD$^2$), a novel methodology that leverages self-data distillation and fine-grained weight sparsity to produce highly efficient and well-aligned draft models. SD$^2$ systematically enhances draft token acceptance rates while significantly reducing Multiply-Accumulate operations (MACs), even in the Universal Assisted Generation (UAG) setting, where draft and target models originate from different model families. On a Llama-3.1-70B target model, SD$^2$ provides a 1.59$\times$ higher Mean Accepted Length (MAL) compared to layer-pruned draft models and reduces MACs by over 43.87% with a 8.36% reduction in MAL compared to a dense draft…

Self-Data Distillation for Recovering Quality in Pruned Large Language Models

Vithursan Thangarasa, Ganesh Venkatesh, Mike Lasby +2 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Large language models have driven significant progress in natural language processing, but their deployment requires substantial compute and memory resources. As models scale, compression techniques become essential for balancing model quality with computational efficiency. Structured pruning, which removes less critical components of the model, is a promising strategy for reducing complexity. However, one-shot pruning often results in significant quality degradation, particularly in tasks requiring multi-step reasoning. To recover lost quality, supervised fine-tuning (SFT) is commonly applied, but it can lead to catastrophic forgetting by shifting the model's learned data distribution. Therefore, addressing the degradation from both pruning and SFT is essential to preserve the original model's quality. In this work, we utilize self-data distilled fine-tuning to address these…

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

Vithursan Thangarasa, Mahmoud Salem, Shreyas Saxena +3 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform general-purpose models in domain-specific tasks (e.g., biomedicine). Although domain-specific pre-training enhances efficiency and leads to smaller models, the computational costs of training these LLMs remain high, posing budgeting challenges. We introduce MediSwift, a suite of biomedical LMs that leverage sparse pre-training on domain-specific biomedical text data. By inducing up to 75% weight sparsity during the pre-training phase, MediSwift achieves a 2-2.5x reduction in training FLOPs. Notably, all sparse pre-training was performed on the Cerebras CS-2 system, which is specifically designed to realize the acceleration benefits from unstructured weight sparsity, thereby significantly enhancing the efficiency…

Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

Vithursan Thangarasa, Shreyas Saxena, Abhay Gupta +1 more

cs.LG · · cited 0× · arXiv ↗ · more like this ·

Recent research has focused on weight sparsity in deep neural network training to reduce FLOPs, aiming for improved efficiency (test accuracy w.r.t training FLOPs). However, sparse weight training often compromises accuracy, requiring extended training schedules to attain the accuracy of dense models. In contrast, our approach, Sparse Iso-FLOP Transformations (Sparse-IFT), uses sparsity to improve accuracy while maintaining dense model FLOPs. Using a single hyperparameter (i.e., the sparsity level), Sparse-IFTs efficiently replace dense layers, expanding the search space for optimal sparse masks. In addition, dynamic sparse training (DST) with Sparse-IFT models effectively navigate this larger sparse mask-weight space, which is evidenced by a spectral analysis using Ramanujan graph properties. Our study reveals a robust correlation among mask topology, weights, and final performance.…

ANovelSystemArchitecture for WebScale Applications Using Lightweight CPUs andVirtualized I/O ∗

Kshitijsudan Saisanthoshbalakrishnan, Sean Lie Minxu, Dhirajmallick Gary +1 more

Information Systems · · cited 0× · OpenAlex ↗ · more like this ·

Large web-scale applicationstypically use a distributed platform, like clusters of commodity servers, to achieve scalable andlow-cost processing. The Map-Reduceframeworkanditsopen-sourceimplementation,Hadoop,iscommonly used to program these applications. Since these applications scale well with an increased number of servers, the cluster size is an important parameter. Cluster size however is constrained by power consumption. In this paper we present a system that uses low-power CPUs to increase the cluster size in a fixed power budget. Using lowpower CPUs leads to the situation where the majority of a server’s power is now consumed by the I/O sub-system. To overcome this, we develop a virtualized I/O sub-system wheremultipleserversshareI/Oresources. AnASICbased high-bandwidth interconnect fabric, and FPGA based I/O cardsimplementthisvirtualizedI/O.Theresultingsystemis the first…

Unbounded Transactional Memory

C. Scott, Ananian Krste, Asanović Bradley +3 more

Computer Networks and Communications · · cited 0× · OpenAlex ↗ · more like this ·

Hardware transactional memory should support un-bounded transactions: transactions of arbitrary size and duration. We describe a hardware implementation of un-bounded transactional memory, called UTM, which ex-ploits the common case for performance without sacri-ficing correctness on transactions whose footprint can be nearly as large as virtual memory. We performed a cycle-accurate simulation of a simplified architecture, called LTM. LTM is based on UTM but is easier to implement, because it does not change the memory subsystem outside of the processor. LTM allows nearly unbounded transac-tions, whose footprint is limited only by physical memory size and whose duration by the length of a timeslice. We assess UTM and LTM through microbenchmark-ing and by automatically converting the SPECjvm98 Java benchmarks and the Linux 2.4.19 kernel to use transac-tions instead of locks. We use both…

1. Background Transactional Memory

Sean Lie

Computer Networks and Communications · · cited 0× · OpenAlex ↗ · more like this ·

Shared memory parallel architectures present a single unified address space to each processor. Usually the memory is physically distributed across the system but each processor is able to access any part of it through a single address space. The system hardware is responsible for presenting this abstraction to each processor. Communication between processors is done implicitly through normal memory load and store operations. For this reason, shared memory architectures generally have much lower processor-to-processor latencies than message passing architectures which often rely on software to handle communication. In addition, the shared memory programming environment is more natural than that of message passing and is better suited for many applications. Often the ability to perform several memory operations atomically is required for correct program execution. Unlike message passing…

TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding

Aditya Sridhar, Nish Sinnadurai, Sean Lie +1 more

cs.LG · · arXiv ↗ · more like this ·

Speculative decoding accelerates LLMs by using a lightweight draft model to generate tokens autoregressively before verifying them in parallel with a larger target model. However, determining the optimal number of tokens to draft remains a key challenge limiting the approach's effectiveness. Dynamic speculative decoding aims to intelligently decide how many tokens to draft to achieve maximum speedups. Existing methods often rely on hand-tuned, sensitive thresholds (e.g., token entropy), which are costly to set and generalize poorly across models and domains. We propose TapOut, an online, training-free, plug-and-play algorithm for dynamic speculation policy selection using multi-armed bandits. Our approach employs a meta-algorithm that selects among multiple parameter-free dynamic speculation strategies based on past reward and exploration. We conduct extensive experiments across diverse…

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Mike Lasby, Ivan Lazarevich, Nish Sinnadurai +3 more

cs.LG · · arXiv ↗ · more like this ·

Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks…

Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment

Abhinav Agarwalla, Abhay Gupta, Alexandre Marques +9 more

cs.CL · · arXiv ↗ · more like this ·

Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs that achieve full accuracy recovery for fine-tuning tasks at up to 70% sparsity. We achieve this for the LLaMA-2 7B model by combining the SparseGPT one-shot pruning method and sparse pretraining of those models on a subset of the SlimPajama dataset mixed with a Python subset of The Stack dataset. We exhibit training acceleration due to sparsity on Cerebras CS-3 chips that closely matches theoretical scaling. In addition, we establish inference acceleration of up to 3x on CPUs by utilizing Neural Magic's DeepSparse engine and 1.7x on GPUs through Neural Magic's nm-vllm engine. The above gains are realized via sparsity alone, thus enabling further gains…