Скачать или смотреть Collective Communication for 100k+ GPUs -- Paper Study

https://arxiv.org/pdf/2510.20171
The NCCLX collective communication framework, developed by Meta, is engineered to optimize the entire lifecycle of large language models (LLMs) by supporting complex workloads on clusters exceeding 100,000 GPUs, addressing the throughput and latency limits faced by traditional communication methods at this scale. Built upon the popular NCCL library, NCCLX introduces a host-driven custom transport layer called CTran, which is fundamentally designed to promote zero-copy and SM-free communication, thereby minimizing GPU resource consumption and avoiding interference with compute overlap schemes. This framework provides a unified communication stack that offers a rich selection of semantics across three execution modes, including Host-initiated APIs and Host-initiated APIs with GPU-resident metadata. Empirical evaluation on the Llama4 model demonstrated that NCCLX achieved substantial communication efficiency improvements, reducing steady training step latency by up to 12% and improving end-to-end decoding latency for inference by 15% to 80%.