an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org
Author
23 papers · 1987–2026
showing 1–23 of 23 RSS
Robert Alverson, Duncan Roweth, Larry Kaplan
The Gemini System Interconnect is a new network for Cray’s supercomputer systems. It provides improved network functionality, latency and issue rate. Latency is reduced with OS bypass for sends and direct user completion notification on receives. Atomic memory operations support the construction of fast synchronization and reduction primitives.
Greg Faanes, Abdulla Bataineh, Duncan Roweth +7 more
Higher global bandwidth requirement for many applications and lower network cost have motivated the use of the Dragonfly network topology for high performance computing systems. In this paper we present the architecture of the Cray Cascade system, a distributed memory system based on the Dragonfly [1] network topology. We describe the structure of the system, its Dragonfly network and the routing algorithms. We describe a set of advanced features supporting both mainstream high performance computing applications and emerging global address space programing models. We present a combination of performance results from prototype systems and simulation data for large systems. We demonstrate the value of the Dragonfly topology and the benefits obtained through extensive use of adaptive routing.
Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon +2 more
The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system size. In this paper, we will describe Slingshot, an interconnection network for large scale computing systems. Slingshot is based on high-radix switches, which allow building exascale and hyperscale datacenters networks with at most three switch-to-switch hops. Moreover, Slingshot provides efficient adaptive routing and congestion control algorithms, and highly tunable traffic classes. Slingshot uses an optimized Ethernet protocol, which allows it to be interoperable with standard Ethernet devices while providing high performance to HPC applications. We analyze the extent to which Slingshot provides these features, evaluating it on microbenchmarks and on several applications from the datacenter and AI worlds, as…
Bruce M. Forrest, Duncan Roweth, N. Stroud +2 more
The remarkable processing capabilities of the nervous system must derive from the large numbers of neurons participating (roughly 1010), since the time-scales involved are of the order of a millisecond, rather than the nanoseconds of modern computers. The neural network models which attempt to capture this behaviour are inherently parallel. We review the implementation of a range of neural network models on SIMD and MIMD computers. On the ICL Distributed Array Processor (DAP), a 4096-processor SIMD machine, we have studied training algorithms in the context of the Hopfield net, with specific applications including the storage of words and continuous text in content-addressable memory. The Hopfield and Tank analogue neural net has been used for image restoration with the Geman and Geman algorithm. We compare the performance of this scheme on the DAP and on a Meiko Computing Surface, a…
Jon Beecroft, David Addison, David Hewson +4 more
QSNET/sup II/ optimizes interprocessor communication in systems built from standard server building blocks. Its short-message processing unit permits fast injection of small messages, providing ultra-low latency and scalability to thousands of nodes. Thus, in a sense, the high-performance network in a cluster computer is the computer because it largely defines achievable performance, widening the range of the applications a cluster can efficiently execute, as well as defining its scalability, fault tolerance, system software, and overall usability.
Dejan Milojičić, Paolo Faraboschi, Nicolas Dubé +1 more
After the end of Dennard scaling and with the imminent end of Moore's Law, it has become challenging to continue scaling HPC systems within a given power envelope. This is exacerbated most in large systems, such as high end supercomputers. To alleviate this problem, general purpose is no longer sufficient, and HPC systems and components are being augmented with special-purpose hardware. By definition, because of the narrow applicability of specialization, broad supercomputing adoption requires using different heterogeneous components, each optimized for a specific application domain. In this paper, we discuss the impact of the introduced heterogeneity of specialization across the HPC stack: interconnects including memory models, accelerators including power and cooling, use cases and applications including AI, and delivery models, such as traditional, as-a-Service, and federated. We…
Daniele De Sensi, Lorenzo Pichetti, Flavio Vella +11 more
Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networks, with bandwidths up to a few terabits per second. However, gauging performance expectations and maximizing system efficiency is challenging due to different technologies, design options, and software layers. This paper comprehensively characterizes three supercomputers - Alps, Leonardo, and LUMI - each with a unique architecture and design. We focus on performance evaluation of intra-node and inter-node interconnects on up to 4096 GPUs, using a mix of intra-node and inter-node benchmarks. By analyzing its limitations and opportunities, we aim to offer practical guidance to researchers, system architects, and software developers dealing with multi-GPU supercomputing. Our results show that there is untapped…
Torsten Hoefler, Ariel Hendel, Duncan Roweth
We discuss the differences and commonalities between network technologies used in supercomputers and data centers and outline a path to convergence at multiple layers. We predict that emerging smart networking solutions will accelerate that convergence.
Salvatore Di Girolamo, Konstantin Taranov, Andreas Kurth +7 more
Applications often communicate data that is non-contiguous in the send- or the receive-buffer, e.g., when exchanging a column of a matrix stored in row-major order. While non-contiguous transfers are well supported in HPC (e.g., MPI derived datatypes), they can still be up to 5x slower than contiguous transfers of the same size. As we enter the era of network acceleration, we need to investigate which tasks to offload to the NIC: In this work we argue that non-contiguous memory transfers can be transparently networkaccelerated, truly achieving zero-copy communications. We implement and extend sPIN, a packet streaming processor, within a Portals 4 NIC SST model, and evaluate strategies for NIC-offloaded processing of MPI datatypes, ranging from datatype-specific handlers to general solutions for any MPI datatype. We demonstrate up to 10x speedup in the unpack throughput of real…
Richard Frederick Barrett, S.D. Hammond, Courtenay Vaughan +4 more
The computing community is in the midst of a disruptive architectural change. The advent of manycore and heterogeneous computing nodes forces us to reconsider every aspect of the system software and application stack. To address this challenge there is a broad spectrum of approaches, which we roughly classify as either revolutionary or evolutionary. With the former, the entire code base is re-written, perhaps using a new programming language or execution model. The latter, which is the focus of this work, seeks a piecewise path of effective incremental change. The end effect of our approach will be revolutionary in that the control structure of the application will be markedly different in order to utilize single-instruction multiple-data/thread (SIMD/SIMT), manycore and heterogeneous nodes, but the physics code fragments will be remarkably similar. Our approach is guided by a set of…
Torsten Hoefler, Duncan Roweth, Keith D. Underwood +9 more
Remote direct memory access (RDMA) over converged Ethernet (RoCE) was an attempt to adopt modern RDMA features into existing Ethernet installations. We revisit RoCE’s design points and conclude that several of its shortcomings must be addressed to fulfill the demands of hyperscale data centers.
Richard Frederick Barrett, Courtenay Vaughan, S.D. Hammond +1 more
For over two decades the dominant means for enabling portable performance of computational science and engineering applications on parallel processing architectures has been the bulk-synchronous parallel programming (BSP) model. Code developers, motivated by performance considerations to minimize the number of messages transmitted, have typically pursued a strategy of aggregating message data into fewer, larger messages. Emerging and future high-performance architectures, especially those seen as targeting Exascale capabilities, provide motivation and capabilities for revisiting this approach. In this paper we explore alternative configurations within the context of a large-scale complex multi-physics application and a proxy that represents its behavior, presenting results that demonstrate some important advantages as the number of processors increases in scale.
In this paper we describe how QsNet/sup II/ supports reduction, a key collective for massively parallel applications. Results from jobs run on a 512-node quad CPU cluster show excellent scaling, with the average time to execute a 2048 process global sum being 22 microsecs.
In this paper we describe QsNet <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">III</sup> , an adaptively routed network for high performance computing (HPC) applications. We detail the structure of the network, the evolution of our adaptive routing algorithms from previous generations of network and new applications of these techniques. We describe other HPC specific features including hardware support for barrier and broadcast and large numbers small packets. We also describe the implementation of the network.
Torsten Hoefler, Duncan Roweth, Keith Underwood +9 more
We observe that emerging artificial intelligence, high-performance computing, and storage workloads pose new challenges for large-scale datacenter networking. RDMA over Converged Ethernet (RoCE) was an attempt to adopt modern Remote Direct Memory Access (RDMA) features into existing Ethernet installations. Now, a decade later, we revisit RoCE's design points and conclude that several of its shortcomings must be addressed to fulfill the demands of hyperscale datacenters. We predict that both the datacenter and high-performance computing markets will converge and adopt modernized Ethernet-based high-performance networking solutions that will replace TCP and RoCE within a decade.
Richard Frederick Barrett, Courtenay Vaughan, S.D. Hammond +1 more
For over two decades the dominant means for enabling portable performance of computational science and engineering applications on parallel processing architectures has been the bulk-synchronous parallel programming model. Code developers, motivated by performance considerations to minimize the number of messages transmitted, have typically strived to increase the size of each message through aggregation strategies. Emerging and future architectures, especially those seen as targeting Exascale capabilities, provide motivation and capabilities for revisiting this approach. In this paper we explore alternative configurations within the context of a large-scale complex multi-physics application and a proxy that represents its behavior, presenting results that demonstrate some important advantages as the number of processors increases in scale.
Tommaso Bonato, Daniele De Sensi, Salvatore Di Girolamo +4 more
Network latency severely impacts the performance of applications running on supercomputers. Adaptive routing algorithms route packets over different available paths to reduce latency and improve network utilization. However, if a switch routes packets belonging to the same network flow on different paths, they might arrive at the destination out-of-order due to differences in the latency of these paths. For some transport protocols like TCP, QUIC, and RoCE, out-of-order (OOO) packets might cause large performance drops or significantly increase CPU utilization. In this work, we propose flowcut switching, a new adaptive routing algorithm that provides high-performance in-order packet delivery. Differently from existing solutions like flowlet switching, which are based on the assumption of bursty traffic and that might still reorder packets, flowcut switching guarantees in-order delivery…
Taylor Groves, Naveen Ravichandrasekaran, Brandon Cook +6 more
Summary Message matching within MPI is an important performance consideration for applications that utilize two‐sided semantics. In this work, we present an instrumentation of the CrayMPI library that allows the collection of detailed message‐matching statistics as well as an implementation of hashed matching in software. We use this functionality to profile key DOE applications with complex communication patterns to determine under what circumstances an application might benefit from hardware offload capabilities within the NIC to accelerate message matching. We find that there are several applications and libraries that exhibit sufficiently long match list lengths to motivate a Binned Message Matching approach.
Abstract: In this paper we study the performance of the all-to-all collective on QsNet II. We compare the effectiveness of a range of algorithms on both small and large element sizes. Results from 1024 node clusters show low latency, a high percentage of peak bandwidth and good scalability.
Duncan Roweth, Ashley Pittman, Jon Beecroft
In this paper we present an in-depth description of how QsNet II supports collectives. Performance data from jobs run on 256-1024 node clusters show that the time to complete barrier synchronization is as low as 5 microseconds, with very good scalability. Results for broadcast indicate that QsNet II can deliver data to 512 nodes in 8-10 microsecs and can sustain an asymptotic bandwidth in excess of 800 Mbytes/sec to all nodes. A 512-node reduction (64-bit floating point sum) completes in 18 microsecs. A gather of a single word from 1024 nodes completes in 30 microsecs. The all-to-all collective delivers 350 Mbytes/sec/node (85 % of peak) across 512-1024 nodes.
QsNetIII and 10 Gbit/s Ethernet, two networks for high perfomance computing. While the proprietary QsNet will continue to provide supercomputing funcionalities to clusters of commodity based servers, the second will establish itself as preferred choice in capacity class systems.Quadrics, who is actively developing products based on the two technologies, will present early performance results and comparisons.
Jörg-Thomas Pfenning, Thomas Pfenning, Zentrum Fur Paralleles Rechnen +4 more
. A interesting feature of some recent parallel computers is the fact that the underlying transport mechanism behind the currently dominating message passing interfaces is based on a global address space model. By accessing this global adress space directly most of the inherent delays for administering message buffers and queues can be avoided. Using this interface we have implemented a user level distributed shared memory layer using the virtual memory protection mechanisms of the operating system. The synchronisation required for maintaining the coherency of the memory is addressed by implemtenting a distributed shared lock which exploits the remote atomic store operations provided by the Meiko CS-2. This allows an asynchonous style of programming where the load is dynamically distributed over the nodes of a parallel partition. Introduction The Meiko CS-2 range of machines are third…
Torsten Hoefler, Mikhail Khalilov, Josiah Clark +9 more
This paper summarizes the opportunities of in-network collective operations (INC) for accelerated collective operations in AI workloads. We provide sufficient detail to make this important field accessible to non-experts in AI or networking, fostering a connection between these communities. Consider two types of INC: Edge-INC, where the system is implemented at the node level, and Core-INC, where the system is embedded within network switches. We outline the potential performance benefits as well as six key obstacles in the context of both Edge-INC and Core-INC that may hinder their adoption. Finally, we present a set of predictions for the future development and application of INC.