arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Artificial Intelligence

Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

Houyang Ge, Fangcheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Y X Wang, Xiaonan Nie, Hailin Zhang, Xupeng Miao, Bin Cui

See on the map

Abstract

Training of large-scale deep learning models necessitates parallelizing the model and data across numerous devices, and the choice of parallelism strategy substantially depends on the training workloads such as memory consumption, computation cost, and communication cost. Current approaches generally assume uniform training workloads across samples in a given task. Thus, existing systems are designed to adopt a static parallelism strategy throughout one training process. Nevertheless, when training models with sequence inputs, this assumption fails due to the sequence length variation across samples. Consequently, training with a static parallelism strategy would result in sub-optimal performance.

Cited by 2

2 papers in the index cite this.

The cited-by list is still filling in while reference lists are collected.

More like this