Accelerating Sharded Data Parallelism at Scale with Federated Learning

Summary

The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence.