arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Julien Herrmann

23 papers · 2013–2026

2013: 22014: 42015: 32016: 32017: 12018: 12019: 22020: 22021: 02022: 12023: 12024: 12025: 12026: 1
2013peak 4 in 20142026

Frequent co-authors

Papers

showing 1–23 of 23 RSS

Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory

Julien Herrmann, Olivier Beaumont, Lionel Eyraud-Dubois +3 more

cs.LG · · cited 50× · arXiv ↗ · more like this ·

This paper introduces a new activation checkpointing method which allows to significantly decrease memory usage when training Deep Neural Networks with the back-propagation algorithm. Similarly to checkpoint-ing techniques coming from the literature on Automatic Differentiation, it consists in dynamically selecting the forward activations that are saved during the training phase, and then automatically recomputing missing activations from those previously recorded. We propose an original computation model that combines two types of activation savings: either only storing the layer inputs, or recording the complete history of operations that produced the outputs (this uses more memory, but requires fewer recomputations in the backward phase), and we provide an algorithm to compute the optimal computation sequence for this model. This paper also describes a PyTorch implementation that…

Acyclic Partitioning of Large Directed Acyclic Graphs

Julien Herrmann, Jonathan Kho, Bora Uçar +2 more

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 41× · DOI ↗ · more like this ·

Finding a good partition of a computational directed acyclic graph associated with an algorithm can help find an execution pattern improving data locality, conduct an analysis of data movement, and expose parallel steps. The partition is required to be acyclic, i.e., the inter-part edges between the vertices from different parts should preserve an acyclic dependency structure among the parts. In this work, we adopt the multilevel approach with coarsening, initial partitioning, and refinement phases for acyclic partitioning of directed acyclic graphs and develop a direct k-way partitioning scheme. To the best of our knowledge, no such scheme exists in the literature. To ensure the acyclicity of the partition at all times, we propose novel and efficient coarsening and refinement heuristics. The quality of the computed acyclic partitions is assessed by computing the edge cut, the total…

A Scalable Clustering-Based Task Scheduler for Homogeneous Processors Using DAG Partitioning

M. Yusuf Özkaya, Anne Benoît, Bora Uçar +2 more

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 39× · DOI ↗ · more like this ·

When scheduling a directed acyclic graph (DAG) of tasks with communication costs on computational platforms, a good trade-off between load balance and data locality is necessary. List-based scheduling techniques are commonly-used greedy approaches for this problem. The downside of list-scheduling heuristics is that they are incapable of making short term sacrifices for the global efficiency of the schedule. In this work, we describe new list-based scheduling heuristics based on clustering for homogeneous platforms, under the realistic duplex single-port communication model. Our approach uses an acyclic partitioner for DAGs for clustering. The clustering enhances the data locality of the scheduler with a global view of the graph. Furthermore, since the partition is acyclic, we can schedule each part completely once its input tasks are ready to be executed. We present an extensive…

Accelerating Linear System Solutions Using Randomization Techniques

Marc Baboulin, Jack Dongarra, Julien Herrmann +1 more

Computational Theory and Mathematics · ACM Transactions on Mathematical Software · · cited 34× · DOI ↗ · more like this ·

We illustrate how linear algebra calculations can be enhanced by statistical techniques in the case of a square linear system Ax = b . We study a random transformation of A that enables us to avoid pivoting and then to reduce the amount of communication. Numerical experiments show that this randomization can be performed at a very affordable computational price while providing us with a satisfying accuracy when compared to partial pivoting. This random transformation called Partial Random Butterfly Transformation (PRBT) is optimized in terms of data storage and flops count. We propose a solver where PRBT and the LU factorization with no pivoting take advantage of the current hybrid multicore/GPU machines and we compare its Gflop/s performance with a solver implemented in a current parallel library.

Optimal Multistage Algorithm for Adjoint Computation

Guillaume Aupy, Julien Herrmann, Paul Hovland +1 more

Computational Theory and Mathematics · SIAM Journal on Scientific Computing · · cited 27× · DOI ↗ · more like this ·

We reexamine the work of Stumm and Walther on multistage algorithms for adjoint computation. We provide an optimal algorithm for this problem when there are two levels of checkpoints, in memory and on disk. Previously, optimal algorithms for adjoint computations were known only for a single level of checkpoints with no writing and reading costs; a well-known example is the binomial checkpointing algorithm of Griewank and Walther. Stumm and Walther extended that binomial checkpointing algorithm to the case of two levels of checkpoints, but they did not provide any optimality results. We bridge the gap by designing the first optimal algorithm in this context. We experimentally compare our optimal algorithm with that of Stumm and Walther to assess the difference in performance.

Optimal memory-aware backpropagation of deep join networks

Olivier Beaumont, Julien Herrmann, Guillaume Pallez +1 more

Computer Vision and Pattern Recognition · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · · cited 20× · DOI ↗ · more like this ·

Deep learning training memory needs can prevent the user from considering large models and large batch sizes. In this work, we propose to use techniques from memory-aware scheduling and automatic differentiation (AD) to execute a backpropagation graph with a bounded memory requirement at the cost of extra recomputations. The case of a single homogeneous chain, i.e. the case of a network whose stages are all identical and form a chain, is well understood and optimal solutions have been proposed in the AD literature. The networks encountered in practice in the context of deep learning are much more diverse, both in terms of shape and heterogeneity. In this work, we define the class of backpropagation graphs, and extend those on which one can compute in polynomial time a solution that minimizes the total number of recomputations. In particular, we consider join graphs which correspond to…

Bridging the Gap between Performance and Bounds of Cholesky Factorization on Heterogeneous Platforms

Emmanuel Agullo, Olivier Beaumont, Lionel Eyraud‐Dubois +4 more

Hardware and Architecture · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 17× · DOI ↗ · more like this ·

We consider the problem of allocating and scheduling dense linear application on fully heterogeneous platforms made of CPUs and GPUs. More specifically, we focus on the Cholesky factorization since it exhibits the main features of such problems. Indeed, the relative performance of CPU and GPU highly depends on the sub-routine: GPUs are for instance much more efficient to process regular kernels such as matrix-matrix multiplications rather than more irregular kernels such as matrix factorization. In this context, one solution consists in relying on dynamic scheduling and resource allocation mechanisms such as the ones provided by PaRSEC or StarPU. In this paper we analyze the performance of dynamic schedulers based on both actual executions and simulations, and we investigate how adding static rules based on an offline analysis of the problem to their decision process can indeed improve…

Periodicity in optimal hierarchical checkpointing schemes for adjoint computations

Guillaume Aupy, Julien Herrmann

Computer Networks and Communications · Optimization methods & software · · cited 12× · DOI ↗ · more like this ·

We reexamine the work of Aupy et al. on optimal algorithms for hierarchical adjoint computations, where two levels of memories are available. The previous optimal algorithm had a quadratic execution time. Here, with structural arguments, namely periodicity, on the optimal solution, we provide an optimal algorithm in constant time and space, with appropriate pre-processing. We also provide an asymptotically optimal algorithm for the online problem, when the adjoint chain size is not known before-hand. Again, these algorithms rely on the proof that the optimal solution for hierarchical adjoint computations is weakly periodic. We conjecture a closed-form formula for the period. Finally, we assess the convergence speed of the approximation ratio for the online problem through simulations.

Memory-Aware List Scheduling for Hybrid Platforms

Julien Herrmann, Loris Marchal, Yves Robert

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 10× · DOI ↗ · more like this ·

This paper provides memory-aware heuristics to schedule tasks graphs onto heterogeneous resources, such as a dual-memory cluster equipped with multicores and a dedicated accelerator (FPGA or GPU). Each task has a different processing time for either resource. The optimization objective is to schedule the graph so as to minimize execution time, given the available memory for each resource type. In addition to ordering the tasks, we must also decide on which resource to execute them, given their computation requirement and the memory currently available on each resource. The major contributions of this paper are twofold: (i) the derivation of an intricate integer linear program formulation for this scheduling problem, and (ii) the design of memory-aware heuristics, which outperform the reference heuristics HEFT and MinMin on a wide variety of problem instances. The absolute performance of…

H-R <scp>evolve</scp>

Julien Herrmann, Guillaume Pallez

Computer Networks and Communications · ACM Transactions on Mathematical Software · · cited 7× · DOI ↗ · more like this ·

We study the problem of checkpointing strategies for adjoint computation on synchronous hierarchical platforms, specifically computational platforms with several levels of storage with different writing and reading costs. When reversing a large adjoint chain, choosing which data to checkpoint and where is a critical decision for the overall performance of the computation. We introduce H-R evolve , an optimal algorithm for this problem. We make it available in a public Python library along with the implementation of several state-of-the-art algorithms for the variant of the problem with two levels of storage. We provide a detailed description of how one can use this library in an adjoint computation software in the field of automatic differentiation or backpropagation. Finally, we evaluate the performance of H-R evolve and other checkpointing heuristics though an extensive campaign of…

Designing LU-QR hybrid solvers for performance and stability

Mathieu Faverge, Julien Herrmann, Julien Langou +3 more

math.NA · · cited 5× · arXiv ↗ · more like this ·

This paper introduces hybrid LU-QR al- gorithms for solving dense linear systems of the form Ax = b. Throughout a matrix factorization, these al- gorithms dynamically alternate LU with local pivoting and QR elimination steps, based upon some robustness criterion. LU elimination steps can be very efficiently parallelized, and are twice as cheap in terms of floating- point operations, as QR steps. However, LU steps are not necessarily stable, while QR steps are always stable. The hybrid algorithms execute a QR step when a robustness criterion detects some risk for instability, and they execute an LU step otherwise. Ideally, the choice between LU and QR steps must have a small computational overhead and must provide a satisfactory level of stability with as few QR steps as possible. In this paper, we introduce several robustness criteria and we establish upper bounds on the growth factor…

Task-based Parallel Programming for Scalable Matrix Product Algorithms

Emmanuel Agullo, Alfredo Buttari, Abdou Guermouche +2 more

Hardware and Architecture · ACM Transactions on Mathematical Software · · cited 2× · DOI ↗ · more like this ·

Task-based programming models have succeeded in gaining the interest of the high-performance mathematical software community because they relieve part of the burden of developing and implementing distributed-memory parallel algorithms in an efficient and portable way.In increasingly larger, more heterogeneous clusters of computers, these models appear as a way to maintain and enhance more complex algorithms. However, task-based programming models lack the flexibility and the features that are necessary to express in an elegant and compact way scalable algorithms that rely on advanced communication patterns. We show that the Sequential Task Flow paradigm can be extended to write compact yet efficient and scalable routines for linear algebra computations. Although, this work focuses on dense General Matrix Multiplication, the proposed features enable the implementation of more complex…

Computing the Expected Makespan of Task Graphs in the Presence of Silent Errors

Henri Casanova, Julien Herrmann, Yves Robert

Hardware and Architecture · · cited 2× · DOI ↗ · more like this ·

Applications structured as Directed Acyclic Graphs (DAGs) of tasks correspond to a general model of parallel computation that occurs in many domains, including popular scientific workflows. DAG scheduling has received an enormous amount of attention, and several list-scheduling heuristics have been proposed and shown to be effective in practice. Many of these heuristics make scheduling decisions based on path lengths in the DAG. At large scale, however, compute platforms and thus tasks are subject to various types of failures with no longer negligible probabilities of occurrence. Failures that have recently received increasing attention are "silent errors," which cause a task to produce incorrect results even though it ran to completion. Tolerating silent errors is done by checking the validity of the results and re-executing the task from scratch in case of an invalid result. The…

Determining the Optimal Redistribution for a Given Data Partition

Thomas Hérault, Julien Herrmann, Loris Marchal +1 more

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 2× · DOI ↗ · more like this ·

The classical redistribution problem aims at optimally scheduling communications when moving from an initial data distribution to a target distribution where each processor will host a subset of data items. However, modern computing platforms are equipped with a powerful interconnection switch, and the cost of a given communication is (almost) independent of the location of its sender and receiver. This leads to generalizing the redistribution problem as follows: find the optimal one-toone mapping of the subsets of data items onto the processors for which the cost of the redistribution is minimal. This paper studies the complexity of this generalized problem. We provide optimal algorithms and evaluate their gain over classical redistribution through simulations. We also show the NP-hardness of the problem to find the optimal data partition and processor permutation (defined by new…

Optimal Re-Materialization Strategies for Heterogeneous Chains: How to Train Deep Neural Networks with Limited Memory

Olivier Beaumont, Lionel Eyraud‐Dubois, Julien Herrmann +2 more

Computer Vision and Pattern Recognition · ACM Transactions on Mathematical Software · · cited 1× · DOI ↗ · more like this ·

Training in Feed Forward Deep Neural Networks is a memory-intensive operation which is usually performed on GPUs with limited memory capacities. This may force data scientists to limit the depth of the models or the resolution of the input data if data does not fit in the GPU memory. The re-materialization technique, whose idea comes from the checkpointing strategies developed in the Automatic Differentiation literature, allows data scientists to limit the memory requirements related to the storage of intermediate data (activations), at the cost of an increase in the computational cost. This paper introduces a new strategy of re-materialization of activations that significantly reduces memory usage. It consists in selecting which activations are saved and which activations are deleted during the forward phase and then recomputing the deleted activations when they are needed during the…

Task-Based Parallel Programming for Scalable Algorithms: application to Matrix Multiplication

Emmanuel Agullo, Alfredo Buttari, Abdou Guermouche +2 more

Hardware and Architecture · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 1× · OpenAlex ↗ · more like this ·

Task-based programming models have succeeded in gaining the interest of the high-performance mathematical software community thanks to how they relieve part of the burden of developing and implementing distributed-memory parallel algorithms in an efficient and portable way. In increasingly larger, more heterogeneous clusters of computers, these models appear as a way to maintain and enhance more complex algorithms. However, task-based programming models lack the flexibility and the features that are necessary to express in an elegant and compact way scalable algorithms that rely on advanced communication patterns. We show that the Sequential Task Flow paradigm can be extended to write a compact yet efficient and scalable General Matrix Multiplication. This extension required few modifications to the StarPU runtime system. The final implementation is shown to be competitive up to 32,768…

Determining the optimal redistribution

Thomas Hérault, Julien Herrmann, Loris Marchal +1 more

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 1× · OpenAlex ↗ · more like this ·

The classical redistribution problem aims at optimally scheduling communications when moving from an initial data distribution \Dini to a target distribution \Dtar where each processor $P_{i}$ will host a subset $P(i)$ of data items. However, modern computing platforms are equipped with a powerful interconnection switch, and the cost of a given communication is (almost) independent of the location of its sender and receiver. This leads to generalizing the redistribution problem as follows: find the optimal permutation $\sigma$ of processors such that $P_{i}$ will host the set $P(\sigma(i))$, and for which the cost of the redistribution is minimal. This report studies the complexity of this generalized problem. We provide optimal algorithms and evaluate their gain over classical redistribution through simulations. We also show the NP-hardness of the problem to find the optimal data…

Tree traversals with task-memory affinities

Julien Herrmann, Loris Marchal, Yves Robert

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 1× · OpenAlex ↗ · more like this ·

We study the complexity of traversing tree-shaped workflows whose tasks require large I/O files. We target a heterogeneous architecture with two resource types, each with a different memory, such as a multicore node equipped with a dedicated accelerator (FPGA or GPU). The tasks in the workflow are colored according to their type and can be processed if all their input and output files can be stored in the corresponding memory. The amount of used memory of each type at a given execution step strongly depends upon the ordering in which the tasks are executed, and upon when communications between both memories are scheduled. The objective is to determine an efficient traversal that minimizes the maximum amount of memory of each type needed to traverse the whole tree. In this paper, we establish the complexity of this two-memory scheduling problem, and provide inapproximability results. In…

Two-level domain-decomposition AdaGrad method for scalable training of graph neural networks

Laurynas Varnas, Julien Herrmann, Alexander Heinlein +2 more

math.NA · · cited 0× · arXiv ↗ · more like this ·

Graph neural networks (GNNs) have emerged as a powerful framework for learning from graph-structured data. However, their efficient training remains challenging, particularly in distributed computing environments. This challenge arises from the use of message passing, which couples all graph nodes, leading to expensive optimization steps, high memory requirements, and substantial communication overhead. To alleviate these limitations, we propose a novel domain-decomposition (DD) variant of AG2m, an AdaGrad method enhanced with second-order curvature information and momentum, denoted by DD-AG2m. The proposed DD-AG2m alternates between AG2m optimization on the original (global) graph and AG2m optimization on the partitioned graphs. To incorporate global information at reduced cost, we further introduce a two-level variant (2DD-AG2m) that performs global optimization steps on a coarse…

A DAG partitioning-assisted list-based scheduler for homogeneous processors

Julien Herrmann, Anne Benoît, Bora Uçar +1 more

Computer Networks and Communications · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 0× · OpenAlex ↗ · more like this ·

When scheduling a directed acyclic graph (DAG) of tasks on computational platforms, a good trade-off between load balance and data locality is necessary. List-based scheduling techniques, such as the earliest finish time (EFT) heuristic, are commonly used greedy approaches for this problem. The downside of EFT, and other list-scheduling heuristics, is that they are incapable of making short-term sacrifices for the global efficiency of the schedule. In this work, we describe three new list-based scheduling heuristics based on clustering for homogeneous platforms. Our approach uses an acyclic partitioner for DAGs for clustering. The clustering enhances the data locality of the scheduler with a global view of the graph. Furthermore, since the partition is acyclic, we can schedule each part completely once its input tasks are ready to be executed. We present an extensive experimental…

Periodicity is Optimal for Offline and Online Multi-Stage Adjoint Computations

Guillaume Aupy, Julien Herrmann

Artificial Intelligence · HAL (Le Centre pour la Communication Scientifique Directe) · · cited 0× · OpenAlex ↗ · more like this ·

We reexamine the work of Aupy et al. on optimal algorithms for multi-stageadjoint computations, where two levels of memories are available. Previousoptimal algorithm had a quadratic execution time. Here, with structuralarguments, namely periodicity, on the optimal solution, we provide an optimalalgorithm in constant time, with appropriate pre-processing. We also provide an asymptotically optimal algorithm for the online problem,when the adjoint graph size is not known before-hand. Again, this algorithmsrely on the proof that the optimal solution for multi-stage adjoint computationsis weakly periodic. We conjecture a closed-form formula for the period. Finally, we experimentallyaccess the convergence speed of the approximation ratio for the online problem,through simulations.

Memory-aware Algorithms and Scheduling Techniques for Matrix Computattions

Julien Herrmann

Hardware and Architecture · theses.fr (ABES) · · cited 0× · OpenAlex ↗ · more like this ·

Throughout this thesis, we have designed memory-aware algorithms and scheduling techniques suitedfor modern memory architectures. We have shown special interest in improving the performance ofmatrix computations on multiple levels. At a high level, we have introduced new numerical algorithmsfor solving linear systems on large distributed platforms. Most of the time, these linear solvers rely onruntime systems to handle resources allocation and data management. We also focused on improving thedynamic schedulers embedded in these runtime systems by adding static information to their decisionprocess. We proposed new memory-aware dynamic heuristics to schedule workflows, that could beimplemented in such runtime systems.Altogether, we have dealt with multiple state-of-the-art factorization algorithms used to solve linearsystems, like the LU, QR and Cholesky factorizations. We targeted…

An In-depth Study of LLM Contributions to the Bin Packing Problem

Julien Herrmann, Guillaume Pallez

cs.AI · · arXiv ↗ · more like this ·

Recent studies have suggested that Large Language Models (LLMs) could provide interesting ideas contributing to mathematical discovery. This claim was motivated by reports that LLM-based genetic algorithms produced heuristics offering new insights into the online bin packing problem under uniform and Weibull distributions. In this work, we reassess this claim through a detailed analysis of the heuristics produced by LLMs, examining both their behavior and interpretability. Despite being human-readable, these heuristics remain largely opaque even to domain experts. Building on this analysis, we propose a new class of algorithms tailored to these specific bin packing instances. The derived algorithms are significantly simpler, more efficient, more interpretable, and more generalizable, suggesting that the considered instances are themselves relatively simple. We then discuss the…