We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Optimizing Single DGX-A100 System: Overcoming GPU Limitations via Efficient Parallelism and Scheduling for Large Language Models.
- Authors
Kim, Kyeong-Hwan; Jeong, Chang-Sung
- Abstract
In this study, we introduce a novel training algorithm specifically designed to overcome the limitations of GPU memory on a single DGX-A100 system. By utilizing the CPU and main memory in the training process and applying a strategy of division and parallelization, our algorithm enhances the size of the trainable language model and the batch size. In addition, we developed a comprehensive management system to effectively manage the execution of the algorithm. This system systematically controls the training process and resource usage, while also enabling the asynchronous deployment of tasks. Finally, we proposed a scheduling technique integrated into the management system, promoting efficient task scheduling in a complex, heterogeneous training environment. These advancements equip researchers with the ability to work with larger models and batch sizes, even when faced with limited GPU memory.
- Subjects
LANGUAGE models; MNEMONICS; NATURAL language processing; SCHEDULING; HETEROGENEOUS computing
- Publication
Applied Sciences (2076-3417), 2023, Vol 13, Issue 16, p9306
- ISSN
2076-3417
- Publication type
Article
- DOI
10.3390/app13169306