CHRONICA: A Data-Imbalance-Aware Scheduler for Distributed Deep Learning

초록

One of the major challenges in distributed deep learning is attenuating straggler problem. The straggler increases synchronization latency and significantly inhibits the convergence of deep learning model. We empirically observe that the imbalanced data samples worsen the straggler problem and make the convergence of the deep learning model slower. However, existing approaches such as BOA and EP4DDL have not addressed data imbalance issues while solving the straggler problem. To overcome the straggler and data imbalance problems, we propose CHRONICA, a new data-imbalance-aware scheduler. Based on the size of the data samples and the configuration of each worker, CHRONICA elaborately predicts the training time required for each worker. CHRONICA then provides equivalent training time to each of the workers, alleviating both step- and epoch-level straggler problems. Furthermore, CHRONICA suggests a new parameter synchronization scheme to achieve fast convergence based on the weighted average of the training workload on each worker. Our extensive evaluation using four deep learning models on 32 Amazon EC2 GPU instances showed that the new CHRONICA achieves up to 3.19 times speedup over the state-of-the-art systems.

키워드