This work introduces an automatic planner that selects the throughput-optimal parallel strategy for training large language models from model and hardware descriptions. It first builds a training-duration simulation model that separates computation, communication, and overlap to estimate step time at the operator level with high accuracy, then uses that cost model to prune about 99% of the parallel-configuration search space before enumerating the survivors. Students learn how an accurate performance model turns an intractable hyperparameter search over data, tensor, and pipeline parallelism into a tractable, globally-optimal one.
Training large transformer language models requires distributing the model and data across many accelerators using combinations of data, tensor, and pipeline parallelism, but choosing the parallel co…