All Reduce Across the Atlantic: Bandwidth in Decentralized Training

The practical realities of devestatingly high communication cost in training.