A Simplified Data Center Network Protocol for Folded-Clos Topology

Datacenters are comprised of thousands of servers, network and storage devices. Data Center Networks (DCNs) are the communications backbone of a datacenter. Several architectural and design innovations have been introduced in DCNs to address the growing size and increasing operational demands of the datacenter. From a protocol perspective, these demands and challenges have been addressed primarily by aggregating multiple off-the-shelf protocols and retrofitting them to the DCN communication needs. This aggregation has resulted in higher overhead, added operational complexity and requires increased effort to perform DCN troubleshooting and maintenance.
In this work we present a new protocol that leverages the structured and symmetrical DCN topology to significantly simplify DCN operations of routing, load balancing, fast failure detection and IP packet forwarding between the servers. We introduce the Multi-Root Meshed Tree Protocol (MR-MTP) which establishes routes without a routing protocol, performs load balancing, provisions fast failure recovery and forwards IP packets between servers. Testing was performed by adopting the folded-Clos topology. The performance of MR-MTP was compared to the popular protocol suite used in folded-Clos topology DCN, i.e. Border Gateway Protocol (BGP for routing), Equal Cost Multipath Protocol (ECMP for load balancing) and Bidirectional Forwarding Detection (BFD to speed up convergence).
As both TCP and UDP are required in folded-Clos (BGP requires TCP for its operation and BFD requires UDP), MR-MTP is replacing six protocols in a DCN router, i.e. BGP, TCP, ECMP, BFD, UDP and IP. MR-MTP is fully backwards-compatible to Internet Protocol (IP) and Ethernet. MR-MTP autoconfigures and auto-assigns routable addresses to the DCN routers, reducing the configuration needs. MR-MTP coded in C language was compared to the protocol suite BGP/ECMP/BFD (from frrouting.org) using folded clos topologies set up in the Fabric testbed (https://portal.fabric-testbed.net).
The results (provided in the slides) of these comparisons clearly demonstrate that significant performance improvement can be achieved with MR-MTP over BGP/ECMP/BFD. The testing evaluated convergence time, control overhead, packets lost, and blast radius on an interface failure. Given its unique approach, MR-MTP offers many other benefits including reduced hardware required to manufacture, immunity from traditional attacks against BGP, TCP and IP, reduced power consumption (and associated cooling costs) among others which will be investigated in the future.

Nirmala Shenoy: Dr. Nirmala Shenoy, received her Bachelors and Master’s in Engineering from Madras University, India. She worked as a Research Scientist in Central Electronics Engineering Research Institute in Chennai, India. She was recipient of the Deutscher Akademischer AustauschDienst, (DAAD) a German fellowship, during which period she received a PhD in Computer Science from University of Bremen, Germany. Dr. Shenoy taught at the Information Communication Institute of Singapore, a collaboration of AT & T Bell Labs US & National Computer Board, Singapore. She subsequently held teaching and research positions in Australian Universities, before joining Rochester Institute of technology, Rochester, New York as Professor in the ISchool, School of Information, Golisano College of Computing and Information Sciences. Her research focus is design, development and evaluation of clean slate protocol solutions to challenging networks problems.

Speaker: Nirmala Shenoy - Rochester Institute of Technology