Author

Shaden Smith

25 papers · 1960–2024

1965peak 4 in 20152024

Frequent co-authors

Papers

showing 1–25 of 25 RSS

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

BigScience Workshop, :, Teven Le Scao +391 more

cs.CL · November 2022 · cited 3,043× · arXiv ↗ · more like this ·

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we…

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Shaden Smith, Mostofa Patwary, Brandon Norick +17 more

cs.CL · January 2022 · cited 870× · arXiv ↗ · more like this ·

Pretrained general-purpose language models can achieve state-of-the-art accuracies in various natural language processing domains by adapting to downstream tasks via zero-shot, few-shot and fine-tuning techniques. Because of their success, the size of these models has increased rapidly, requiring high-performance hardware, software, and algorithmic techniques to enable training such large models. As the result of a joint effort between Microsoft and NVIDIA, we present details on the training of the largest monolithic transformer based language model, Megatron-Turing NLG 530B (MT-NLG), with 530 billion parameters. In this paper, we first focus on the infrastructure as well as the 3D parallelism methodology used to train this model using DeepSpeed and Megatron. Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a…

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang +8 more

cs.LG · 30 Jun 2022 · cited 714× · arXiv ↗ · more like this ·

The past several years have witnessed the success of transformer-based models, and their scale and application scenarios continue to grow aggressively. The current landscape of transformer models is increasingly diverse: the model size varies drastically with the largest being of hundred-billion parameters; the model characteristics differ due to the sparsity introduced by the Mixture-of-Experts; the target application scenarios can be latency-critical or throughput-oriented; the deployment hardware could be single- or multi-GPU systems with different types of memory and storage, etc. With such increasing diversity and the fast-evolving pace of transformer models, designing a highly performant and efficient inference system is extremely challenging. In this paper, we present DeepSpeed Inference, a comprehensive system solution for transformer model inference to address the…

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley +2 more

cs.DC · 16 Apr 2021 · cited 577× · arXiv ↗ · more like this ·

In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model. In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale…

ZeRO-infinity

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley +2 more

Hardware and Architecture · 21 Oct 2021 · cited 219× · DOI ↗ · more like this ·

SPLATT: Efficient and Parallel Sparse Tensor-Matrix Multiplication

Shaden Smith, Niranjay Ravindran, Nicholas D. Sidiropoulos +1 more

Computational Mathematics · University of Minnesota Digital Conservancy (University of Minnesota) · May 2015 · cited 218× · DOI ↗ · more like this ·

Multi-dimensional arrays, or tensors, are increasingly found in fields such as signal processing and recommender systems. Real-world tensors can be enormous in size and often very sparse. There is a need for efficient, high-performance tools capable of processing the massive sparse tensors of today and the future. This paper introduces SPLATT, a C library with shared-memory parallelism for three-mode tensors. SPLATT contains algorithmic improvements over competing state of the art tools for sparse tensor factorization. SPLATT has a fast, parallel method of multiplying a matricide tensor by a Khatri-Rao product, which is a key kernel in tensor factorization methods. SPLATT uses a novel data structure that exploits the sparsity patterns of tensors. This data structure has a small memory footprint similar to competing methods and allows for the computational improvements featured in our…

Tensor-matrix products with a compressed sparse tensor

Shaden Smith, George Karypis

Computational Mathematics · University of Minnesota Digital Conservancy (University of Minnesota) · 15 Nov 2015 · cited 133× · DOI ↗ · more like this ·

The Canonical Polyadic Decomposition (CPD) of tensors is a powerful tool for analyzing multi-way data and is used extensively to analyze very large and extremely sparse datasets. The bottleneck of computing the CPD is multiplying a sparse tensor by several dense matrices. Algorithms for tensor-matrix products fall into two classes. The first class saves floating point operations by storing a compressed tensor for each dimension of the data. These methods are fast but suffer high memory costs. The second class uses a single uncompressed tensor at the cost of additional floating point operations. In this work, we bridge the gap between the two approaches and introduce the compressed sparse fiber (CSF) a data structure for sparse tensors along with a novel parallel algorithm for tensor-matrix multiplication. CSF offers similar operation reductions as existing compressed methods while using…

Tensaurus: A Versatile Accelerator for Mixed Sparse-Dense Tensor Computations

Nitish Srivastava, Hanchen Jin, Shaden Smith +3 more

Computational Mathematics · February 2020 · cited 120× · DOI ↗ · more like this ·

Tensor factorizations are powerful tools in many machine learning and data analytics applications. Tensors are often sparse, which makes sparse tensor factorizations memory bound. In this work, we propose a hardware accelerator that can accelerate both dense and sparse tensor factorizations. We co-design the hardware and a sparse storage format, which allows accessing the sparse data in vectorized and streaming fashion and maximizes the utilization of the memory bandwidth. We extract a common computation pattern that is found in numerous matrix and tensor operations and implement it in the hardware. By designing the hardware based on this common compute pattern, we can not only accelerate tensor factorizations but also mixed sparse-dense matrix operations. We show significant speedup and energy benefit over the state-of-the-art CPU and GPU implementations of tensor factorizations and…

Bridging the gap between HPC and big data frameworks

Michael Anderson, Shaden Smith, Narayanan Sundaram +5 more

Information Systems · Proceedings of the VLDB Endowment · April 2017 · cited 48× · DOI ↗ · more like this ·

Apache Spark is a popular framework for data analytics with attractive features such as fault tolerance and interoperability with the Hadoop ecosystem. Unfortunately, many analytics operations in Spark are an order of magnitude or more slower compared to native implementations written with high performance computing tools such as MPI. There is a need to bridge the performance gap while retaining the benefits of the Spark ecosystem such as availability, productivity, and fault tolerance. In this paper, we propose a system for integrating MPI with Spark and analyze the costs and benefits of doing so for four distributed graph and machine learning applications. We show that offloading computation to an MPI environment from within Spark provides 3.1−17.7× speedups on the four sparse applications, including all of the overheads. This opens up an avenue to reuse existing MPI libraries in…

Streaming Tensor Factorization for Infinite Data Sources

Shaden Smith, Kejun Huang, Nicholas D. Sidiropoulos +1 more

Computational Mathematics · Society for Industrial and Applied Mathematics eBooks · 7 May 2018 · cited 44× · DOI ↗ · more like this ·

Sparse tensor factorization is a popular tool in multi-way data analysis and is used in applications such as cybersecurity, recommender systems, and social network analysis. In many of these applications, the tensor is not known a priori and instead arrives in a streaming fashion for a potentially unbounded amount of time. Existing approaches for streaming sparse tensors are not practical for unbounded streaming because they rely on maintaining the full factorization of the data, which grows linearly with time. In this work, we present CP-stream, an algorithm for streaming factorization in the model of the canonical polyadic decomposition which does not grow linearly in time or space, and is thus practical for long-term streaming. Additionally, CP-stream incorporates user-specified constraints such as non-negativity which aid in the stability and interpretability of the factorization.…

A Medium-Grained Algorithm for Sparse Tensor Factorization

Shaden Smith, George Karypis

Computational Mathematics · May 2016 · cited 44× · DOI ↗ · more like this ·

Modeling multi-way data can be accomplished using tensors, which are data structures indexed along three or more dimensions. Tensors are increasingly used to analyze extremely large and sparse multi-way datasets in life sciences, engineering, and business. The canonical polyadic decomposition (CPD) is a popular tensor factorization for discovering latent features and is most commonly found via the method of alternating least squares (CPD-ALS). The computational time and memory required to compute CPD limits the size and dimensionality of the tensors that can be solved on a typical workstation, making distributed solution approaches the only viable option. Most methods for distributed-memory systems have focused on distributing the tensor in a coarse-grained, one-dimensional fashion that prohibitively requires the dense matrix factors to be fully replicated on each node. Recent work…

Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory

Shaden Smith, Jongsoo Park, George Karypis

Computational Mathematics · University of Minnesota Digital Conservancy (University of Minnesota) · May 2017 · cited 36× · DOI ↗ · more like this ·

HPC systems are increasingly used for data intensive computations which exhibit irregular memory accesses, non-uniform work distributions, large memory footprints, and high memory bandwidth demands. To address these challenging demands, HPC systems are turning to many-core architectures that feature a large number of energy-efficient cores backed by high-bandwidth memory. These features are exemplified in Intel's recent Knights Landing many-core processor (KNL), which typically has 68 cores and 16GB of on-package multi-channel DRAM (MCDRAM). This work investigates how the novel architectural features offered by KNL can be used in the context of decomposing sparse, unstructured tensors using the canonical polyadic decomposition (CPD). The CPD is used extensively to analyze large multi-way datasets arising in various areas including precision healthcare, cybersecurity, and e-commerce.…

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization

Shaden Smith, George Karypis

Computational Mathematics · University of Minnesota Digital Conservancy (University of Minnesota) · 2 May 2016 · cited 33× · OpenAlex ↗ · more like this ·

Modeling multi-way data can be accomplished using tensors, which are data structures \n indexed along three or more dimensions. Tensors are increasingly used to analyze \n extremely large and sparse multi-way datasets in life sciences, engineering, and business. \n The canonical polyadic decomposition (CPD) is a popular tensor factorization for \n discovering latent features and is most commonly found via the method of alternating \n least squares (CPD-ALS). The computational time and memory required to compute CPD \n limits the size and dimensionality of the tensors that can be solved on a typical \n workstation, making distributed solution approaches the only viable option. \n Most methods for distributed-memory systems have focused on distributing the tensor \n in a coarse-grained, one-dimensional fashion that prohibitively requires the dense \n matrix factors to be fully replicated…

Memory-efficient parallel computation of tensor and matrix products for big tensor decomposition

Niranjay Ravindran, Nicholas D. Sidiropoulos, Shaden Smith +1 more

Computational Mathematics · 2014 48th Asilomar Conference on Signals, Systems and Computers · November 2014 · cited 33× · DOI ↗ · more like this ·

Low-rank tensor decomposition has many applications in signal processing and machine learning, and is becoming increasingly important for analyzing big data. A significant challenge is the computation of intermediate products which can be much larger than the final result of the computation, or even the original tensor. We propose a scheme that allows memory-efficient in-place updates of intermediate matrices. Motivated by recent advances in big tensor decomposition from multiple compressed replicas, we also consider the related problem of memory-efficient tensor compression. The resulting algorithms can be parallelized, and can exploit but do not require sparsity.

Blocking Optimization Techniques for Sparse Tensor Computation

Jee Choi, Xing Liu, Shaden Smith +1 more

Computational Mathematics · Maryland Shared Open Access Repository (USMAI Consortium) · May 2018 · cited 31× · DOI ↗ · more like this ·

We present a detailed analysis of the sparse matricized tensor times Khatri-Rao product (MTTKRP) kernel that is the key bottleneck in various sparse tensor computations. By using the well-known roofline model and carefully instrumenting the state-of-the-art MTTKRP code with the pressure point analysis technique, we show that the performance of MTTKRP is highly sensitive to data traffic like other sparse computations. We also identify key performance bottlenecks that diverge from our prior knowledge of sparse MTTKRP on modern processors. We propose to use blocking optimization techniques to address the bottlenecks identified within the MTTKRP computation. By using a combination of two blocking techniques, we achieve more than 3.5× and 2.0× speedup over the state-of-the-art MTTKRP implementation in the SPLATT library on four real-world data sets and two synthetically generated data sets…

Constrained Tensor Factorization with Accelerated AO-ADMM

Shaden Smith, Alec Beri, George Karypis

Computational Mathematics · August 2017 · cited 28× · DOI ↗ · more like this ·

Low-rank sparse tensor factorization is a popular tool for analyzing multi-way data and is used in domains such as recommender systems, precision healthcare, and cybersecurity. Imposing constraints on a factorization, such as non-negativity or sparsity, is a natural way of encoding prior knowledge of the multi-way data. While constrained factorizations are useful for practitioners, they can greatly increase factorization time due to slower convergence and computational overheads. Recently, a hybrid of alternating optimization and alternating direction method of multipliers (AO-ADMM) was shown to have both a high convergence rate and the ability to naturally incorporate a variety of popular constraints. In this work, we present a parallelization strategy and two approaches for accelerating AO-ADMM. By redefining the convergence criteria of the inner ADMM iterations, we are able to split…

An Exploration of Optimization Algorithms for High Performance Tensor Completion

Shaden Smith, Jongsoo Park, George Karypis

Computational Mathematics · November 2016 · cited 21× · DOI ↗ · more like this ·

Tensor completion is a powerful tool used to estimate or recover missing values in multi-way data. It has seen great success in domains such as product recommendation and healthcare. Tensor completion is most often accomplished via low-rank sparse tensor factorization, a computationally expensive non-convex optimization problem which has only recently been studied in the context of parallel computing. In this work, we study three optimization algorithms that have been successfully applied to tensor completion: alternating least squares (ALS), stochastic gradient descent (SGD), and coordinate descent (CCD++). We explore opportunities for parallelism on shared- and distributed-memory systems and address challenges such as memory- and operation-efficiency, load balance, cache locality, and communication. Among our advancements are an SGD algorithm which combines stratification with…

Big Data and Recommender Systems

David C. Anastasiu, Evangelia Christakopoulou, Shaden Smith +2 more

Information Systems · University of Minnesota Digital Conservancy (University of Minnesota) · 2016 · cited 9× · DOI ↗ · more like this ·

Recommender systems are ubiquitous in today's marketplace and have great commercial importance, as evidenced by the large number of companies that sell recommender systems solutions. Successful recommender systems use past product purchase and satisfaction data to make high quality personalized recommendations. The vast amounts of data available to recommender systems today forces a total re-evaluation of the methods used to compute recommendations. In this paper, we provide an overview of recommender systems in the era of Big Data. We highlight prevailing recommendation algorithms and how they have been adapted to operate in parallel and distributed computing environments. Within the recommender systems context, we focus our discussion on two specific challenges: how to scale up finding nearest neighbors and how to scale latent factor recommendation methods.

Scalable Label Propagation for Multi-relational Learning on the Tensor Product of Graphs

Zhuliu Li, Raphael Petegrosso, Shaden Smith +3 more

cs.LG · February 2018 · cited 7× · arXiv ↗ · more like this ·

Multi-relational learning on knowledge graphs infers high-order relations among the entities across the graphs. This learning task can be solved by label propagation on the tensor product of the knowledge graphs to learn the high-order relations as a tensor. In this paper, we generalize a widely used label propagation model to the normalized tensor product graph, and propose an optimization formulation and a scalable Low-rank Tensor-based Label Propagation algorithm (LowrankTLP) to infer multi-relations for two learning tasks, hyperlink prediction and multiple graph alignment. The optimization formulation minimizes the upper bound of the noisy tensor estimation error for multiple graph alignment, by learning with a subset of the eigen-pairs in the spectrum of the normalized tensor product graph. We also provide a data-dependent transductive Rademacher bound for binary hyperlink…

DMS: Distributed Sparse Tensor Factorization with Alternating Least Squares

Shaden Smith, George Karypis

Computational Mathematics · University of Minnesota Digital Conservancy (University of Minnesota) · 12 May 2015 · cited 7× · OpenAlex ↗ · more like this ·

Tensors are data structures indexed along three or more dimensions. Tensors have found increasing use in domains such as data mining and recommender systems where dimensions can have enormous length and are resultingly very sparse. The canonical polyadic decomposition (CPD) is the most popular tensor factorization for discovering latent features and is most commonly found via the method of alternating least squares (CPD-ALS). Factoring large, sparse tensors is a computationally challenging task which can no longer be done in the memory of a typical workstation. State of the art methods for distributed memory systems have focused on decomposing the tensor in a one-dimensional (1D) fashion that prohibitively requires the dense matrix factors to be fully replicated on each node. To that effect, we present DMS, a novel distributed CPD-ALS algorithm. DMS utilizes a 3D decomposition that…

High Performance Streaming Tensor Decomposition

Yongseok Soh, Patrick Flick, Xing Liu +4 more

Computational Mathematics · May 2021 · cited 5× · DOI ↗ · more like this ·

We present a new algorithm for computing tensor decomposition on streaming data that achieves up to 102× speedup over the state-of-the-art CP-stream algorithm through lower computational complexity and performance optimization. For each streaming time slice, our algorithm partitions the factor matrix rows into those with and without updates and keeps them in Gram matrix form to significantly reduce the required computation. We also improve the scalability and performance of the matricized tensor times Khatri-Rao product (MTTKRP) kernel, a key performance bottleneck in many tensor decomposition algorithms, by reducing the synchronization overhead through the combined use of mutex locks and thread-local memory. For problems with constraints (e.g., non-negativity), we apply data blocking and operation fusion to the alternating direction method of multiplier (ADMM) kernel in the constrained…

HEAT: A Highly Efficient and Affordable Training System for Collaborative Filtering Based Recommendation on CPUs

Chengming Zhang, Shaden Smith, Baixi Sun +6 more

cs.DC · April 2023 · cited 1× · arXiv ↗ · more like this ·

Collaborative filtering (CF) has been proven to be one of the most effective techniques for recommendation. Among all CF approaches, SimpleX is the state-of-the-art method that adopts a novel loss function and a proper number of negative samples. However, there is no work that optimizes SimpleX on multi-core CPUs, leading to limited performance. To this end, we perform an in-depth profiling and analysis of existing SimpleX implementations and identify their performance bottlenecks including (1) irregular memory accesses, (2) unnecessary memory copies, and (3) redundant computations. To address these issues, we propose an efficient CF training system (called HEAT) that fully enables the multi-level caching and multi-threading capabilities of modern CPUs. Specifically, the optimization of HEAT is threefold: (1) It tiles the embedding matrix to increase data locality and reduce cache…

RecFlex: Enabling Feature Heterogeneity-Aware Optimization for Deep Recommendation Models with Flexible Schedules

Zaifeng Pan, Zheng Zhen, Feng Zhang +7 more

Information Systems · 17 Nov 2024 · cited 0× · DOI ↗ · more like this ·

Industrial recommendation models typically involve numerous feature fields. The embedding computation workloads are heterogeneous across these fields, thus requiring varied optimal code schedules. While existing solutions apply basic fusion optimization for embedding operations, they inefficiently treat all feature fields with identical schedules, leading to suboptimal performance. In this paper, we introduce RecFlex, which generates fused kernels with distinct schedules for different feature fields. RecFlex employs the interference-aware schedule tuner to tune schedules and the heterogeneous schedule fusion compiler to generate fused kernels, addressing two major challenges. To determine optimal schedules of different feature fields within the fused kernel, RecFlex proposes a two-stage interferencesimulated tuning strategy. To handle dynamic workloads that challenge tuning and fusion…

Powered by TCPDF (www.tcpdf.org) DMS: Distributed Sparse Tensor Factorization with Alternating Least Squares

Shaden Smith, George Karypis, Shaden Smith +1 more

Computational Mathematics · 2015 · cited 0× · OpenAlex ↗ · more like this ·

Abstract—Tensors are data structures indexed along three or more dimensions. Tensors have found increasing use in domains such as data mining and recommender systems where dimensions can have enormous length and are resultingly very sparse. The canonical polyadic decomposition (CPD) is the most popular tensor factorization for discovering latent features and is most commonly found via the method of alternating least squares (CPD-ALS). Factoring large, sparse tensors is a computationally challenging task which can no longer be done in the memory of a typical workstation. State of the art methods for distributed memory systems have focused on decomposing the tensor in a one-dimensional (1D) fashion that prohibitively requires the dense matrix factors to be fully replicated on each node. To that effect, we present DMS, a novel distributed CPD-ALS algorithm. DMS utilizes a 3D decomposition…

A POWER-SPECTRUM PROGRAM FOR THE IBM 650 COMPUTER

Shaden Smith

Computational Theory and Mathematics · Defense Technical Information Center (DTIC) · 15 Oct 1960 · cited 0× · OpenAlex ↗ · more like this ·

The power-spectrum program described in this report uses for its input the autocorrelation function, as calculated by the program described in DRL AF Technical Memorandum No. 51. The spectrum program prepares a table of the cosines which will be needed and then generates appropriate look-up instructions for the cosines during the computation. This is considerably faster than recomputing each cosine as needed. The output of the spectrum program gives the frequency interval, the raw spectrum, the Hammed and Hanned spectra,AND THE CUMULATIVE OR INTEGRATED VERSIONS OF THESE THREE SPECTRA. (Author)