arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Tim Moon

16 papers · 2016–2026

2016: 22017: 02018: 12019: 32020: 02021: 42022: 12023: 02024: 02025: 32026: 2
2016peak 4 in 20212026

Frequent co-authors

Papers

showing 1–16 of 16 RSS

Communication quantization for data-parallel training of deep neural networks

Nikoli Dryden, Sam Adé Jacobs, Tim Moon +1 more

Computer Vision and Pattern Recognition · IEEE International Conference on High Performance Computing, Data, and Analytics · · cited 122× · DOI ↗ · more like this ·

We study data-parallel training of deep neural networks on high-performance computing infrastructure. The key problem with scaling data-parallel training is avoiding severe communication/computation imbalance. We explore quantizing gradient updates before communication to reduce bandwidth requirements and compare it against a baseline implementation that uses the MPI allreduce routine. We port two existing quantization approaches, one-bit and threshold, and develop our own adaptive quantization algorithm. The performance of these algorithms is evaluated and compared with MPI_Allreduce when training models for the MNIST dataset and on a synthetic benchmark. On an HPC system, MPI_Allreduce outperforms the existing quantization approaches. Our adaptive quantization is comparable or superior for large layers without sacrificing accuracy. It is 1.76 times faster than the next best approach…

Channel and filter parallelism for large-scale CNN training

Nikoli Dryden, Naoya Maruyama, Tim Moon +3 more

Computer Vision and Pattern Recognition · · cited 44× · DOI ↗ · more like this ·

Accelerating large-scale CNN training is needed to keep training times reasonable as datasets grow larger and models become more complex. Existing frameworks primarily scale using data-parallelism, but this is limited by the mini-batch size, which cannot grow arbitrarily. We introduce three algorithms that partition channel or filter data to exploit parallelism beyond the sample dimension. Further, they partition the parameters of convolutional layers, replacing global all reduces with segmented allreduces---smaller, concurrent allreduces among disjoint processor sets. These algorithms enable strong scaling, reduced communication overhead, and reduced memory pressure, enabling training of very wide CNNs.

Aluminum: An Asynchronous, GPU-Aware Communication Library Optimized for Large-Scale Training of Deep Neural Networks on HPC Systems

Nikoli Dryden, Naoya Maruyama, Tim Moon +4 more

Artificial Intelligence · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · · cited 42× · DOI ↗ · more like this ·

The following topics are dealt with: learning (artificial intelligence); parallel processing; neural nets; convolutional neural nets; data analysis; stochastic processes; gradient methods; image classification; message passing; evolutionary computation.

Enabling rapid COVID-19 small molecule drug design through scalable deep learning of generative models

Sam Adé Jacobs, Tim Moon, Kevin McLoughlin +9 more

Computational Theory and Mathematics · The International Journal of High Performance Computing Applications · · cited 37× · DOI ↗ · more like this ·

We improved the quality and reduced the time to produce machine learned models for use in small molecule antiviral design. Our globally asynchronous multi-level parallel training approach strong scales to all of Sierra with up to 97.7% efficiency. We trained a novel, character-based Wasserstein autoencoder that produces a higher quality model trained on 1.613 billion compounds in 23 minutes while the previous state of the art takes a day on 1 million compounds. Reducing training time from a day to minutes shifts the model creation bottleneck from computer job turnaround time to human innovation time. Our implementation achieves 318 PFLOPs for 17.1% of half-precision peak. We will incorporate this model into our molecular design loop enabling the generation of more diverse compounds; searching for novel, candidate antiviral drugs improves and reduces the time to synthesize compounds to…

SUPER: SUb-Graph Parallelism for TransformERs

Arpan Jain, Tim Moon, Tom Benson +4 more

Artificial Intelligence · · cited 9× · DOI ↗ · more like this ·

Transformer models have revolutionized the field of Natural Language Processing (NLP) and they achieve state-of-the-art performance in applications like machine translation, question answering, regression, and summarization. However, training Transformers is challenging because of their large memory and compute requirements. The literature contains several approaches to parallelize training, like layer parallelism and pipeline parallelism, but they are optimized to benefit out-of-core models and they don't exploit the inherent parallelism in Transformer models. Other work uses model parallelism to achieve weak scaling by increasing the model size. In this paper, we propose sub-graph parallelism that provides a significant performance improvement over pure data parallelism with a fixed number of resources, and as an additional technique for strong- and weak-scaling without increasing…

Learning Interpretable Models Through Multi-Objective Neural Architecture Search

Zachariah Carmichael, Tim Moon, Sam Ade Jacobs

cs.LG · · cited 3× · arXiv ↗ · more like this ·

Monumental advances in deep learning have led to unprecedented achievements across various domains. While the performance of deep neural networks is indubitable, the architectural design and interpretability of such models are nontrivial. Research has been introduced to automate the design of neural network architectures through neural architecture search (NAS). Recent progress has made these methods more pragmatic by exploiting distributed computation and novel optimization algorithms. However, there is little work in optimizing architectures for interpretability. To this end, we propose a multi-objective distributed NAS framework that optimizes for both task performance and "introspectability," a surrogate metric for aspects of interpretability. We leverage the non-dominated sorting genetic algorithm (NSGA-II) and explainable AI (XAI) techniques to reward architectures that can be…

Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelism

Nikoli Dryden, Naoya Maruyama, Tom Benson +3 more

cs.DC · · cited 3× · arXiv ↗ · more like this ·

Scaling CNN training is necessary to keep up with growing datasets and reduce training time. We also see an emerging need to handle datasets with very large samples, where memory requirements for training are large. Existing training frameworks use a data-parallel approach that partitions samples within a mini-batch, but limits to scaling the mini-batch size and memory consumption makes this untenable for large samples. We describe and implement new approaches to convolution, which parallelize using spatial decomposition or a combination of sample and spatial decomposition. This introduces many performance knobs for a network, so we develop a performance model for CNNs and present a method for using it to automatically determine efficient parallelization strategies. We evaluate our algorithms with microbenchmarks and image classification with ResNet-50. Our algorithms allow us to…

Accelerating eigenvector and pseudospectra computation using blocked multi-shift triangular solves

Tim Moon, Jack Poulson

cs.MS · · cited 3× · arXiv ↗ · more like this ·

Multi-shift triangular solves are basic linear algebra calculations with applications in eigenvector and pseudospectra computation. We propose blocked algorithms that efficiently exploit Level 3 BLAS to perform multi-shift triangular solves and safe multi-shift triangular solves. Numerical experiments indicate that computing triangular eigenvectors with a safe multi-shift triangular solve achieves speedups by a factor of 60 relative to LAPACK. This algorithm accelerates the calculation of general eigenvectors threefold. When using multi-shift triangular solves to compute pseudospectra, we report ninefold speedups relative to EigTool.

Parallelizing Graph Neural Networks via Matrix Compaction for Edge-Conditioned Networks

Shehtab Zaman, Tim Moon, Tom Benson +3 more

Artificial Intelligence · 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid) · · cited 1× · DOI ↗ · more like this ·

Graph neural networks (GNNs) are a powerful approach for machine learning on graph datasets. Such datasets often consist of millions of modestly-sized graphs, making them well-suited for data-parallel training. However, existing methods show poor scaling due to load imbalances and kernel overheads. We propose an optimized 2D scatter-gather based represen-tation of GNNs that is amenable to distributed, data-parallel training without changing the underlying mathematics of the GNN. By padding graph data to a fixed size on each process, we can simplify data ingestion, make use of efficient compute kernels, equally distribute computation load, and reduce overheads. We benchmark edge-conditioned GNNs with the PCQM4M-LSC and OGB-PPA datasets. Our implementation shows better runtime performance than the state-of-the-art, with a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML"…

Pretraining Large Language Models with NVFP4

NVIDIA, Felix Abecassis, Anjulie Agrusa +87 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improvements in computational speed and resource utilization. However, quantization at this level poses challenges to training stability, convergence, and implementation, notably for large-scale models…

NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model

NVIDIA, :, Aarti Basant +214 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

We introduce Nemotron-Nano-9B-v2, a hybrid Mamba-Transformer language model designed to increase throughput for reasoning workloads while achieving state-of-the-art accuracy compared to similarly-sized models. Nemotron-Nano-9B-v2 builds on the Nemotron-H architecture, in which the majority of the self-attention layers in the common Transformer architecture are replaced with Mamba-2 layers, to achieve improved inference speed when generating the long thinking traces needed for reasoning. We create Nemotron-Nano-9B-v2 by first pre-training a 12-billion-parameter model (Nemotron-Nano-12B-v2-Base) on 20 trillion tokens using an FP8 training recipe. After aligning Nemotron-Nano-12B-v2-Base, we employ the Minitron strategy to compress and distill the model with the goal of enabling inference on up to 128k tokens on a single NVIDIA A10G GPU (22GiB of memory, bfloat16 precision). Compared to…

Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

Sam Ade Jacobs, Brian Van Essen, David Hysom +11 more

cs.DC · · cited 0× · arXiv ↗ · more like this ·

Training deep neural networks on large scientific data is a challenging task that requires enormous compute power, especially if no pre-trained models exist to initialize the process. We present a novel tournament method to train traditional as well as generative adversarial networks built on LBANN, a scalable deep learning framework optimized for HPC systems. LBANN combines multiple levels of parallelism and exploits some of the worlds largest supercomputers. We demonstrate our framework by creating a complex predictive model based on multi-variate data from high-energy-density physics containing hundreds of millions of images and hundreds of millions of scalar values derived from tens of millions of simulations of inertial confinement fusion. Our approach combines an HPC workflow and extends LBANN with optimized data ingestion and the new tournament-style training algorithm to produce…

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

NVIDIA, :, Aaron Blakeman +571 more

cs.CL · · arXiv ↗ · more like this ·

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We…

Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

NVIDIA, :, Aakshita Chandiramani +544 more

cs.LG · · arXiv ↗ · more like this ·

We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along…

NVIDIA Nemotron 3: Efficient and Open Intelligence

NVIDIA, :, Aaron Blakeman +356 more

cs.CL · · arXiv ↗ · more like this ·

We introduce the Nemotron 3 family of models - Nano, Super, and Ultra. These models deliver strong agentic, reasoning, and conversational capabilities. The Nemotron 3 family uses a Mixture-of-Experts hybrid Mamba-Transformer architecture to provide best-in-class throughput and context lengths of up to 1M tokens. Super and Ultra models are trained with NVFP4 and incorporate LatentMoE, a novel approach that improves model quality. The two larger models also include MTP layers for faster text generation. All Nemotron 3 models are post-trained using multi-environment reinforcement learning enabling reasoning, multi-step tool use, and support granular reasoning budget control. Nano, the smallest model, outperforms comparable models in accuracy while remaining extremely cost-efficient for inference. Super is optimized for collaborative agents and high-volume workloads such as IT ticket…

HARM3D+NUC: A new method for simulating the post-merger phase of binary neutron star mergers with GRMHD, tabulated EOS and neutrino leakage

Ariadna Murguia-Berthier, Scott C. Noble, Luke F. Roberts +25 more

astro-ph.HE · · arXiv ↗ · more like this ·

The first binary neutron star merger has already been detected in gravitational waves. The signal was accompanied by an electromagnetic counterpart including a kilonova component powered by the decay of radioactive nuclei, as well as a short $γ$-ray burst. In order to understand the radioactively-powered signal, it is necessary to simulate the outflows and their nucleosynthesis from the post-merger disk. Simulating the disk and predicting the composition of the outflows requires general relativistic magnetohydrodynamical (GRMHD) simulations that include a realistic, finite-temperature equation of state (EOS) and self-consistently calculating the impact of neutrinos. In this work, we detail the implementation of a finite-temperature EOS and the treatment of neutrinos in the GRMHD code HARM3D+NUC, based on HARM3D. We include formal tests of both the finite-temperature EOS and the neutrino…