RNG: Flat Datacenter Networks at Scale

Topology

1. Physical and Logical StructureFlat Expander Graph (Quasi-Random): RNG removes intermediate aggregation and spine layers entirely. It connects Top-of-Rack (ToR) routers directly to each other in a flat, quasi-random topology, forming an optimal expander graph. All ToRs (Top-of-Rack switches) act as peers directly connected to each other. There are no dedicated spine or aggregation switches.
2. Breakout Lanes: Physical router ports (e.g., 400 Gbps) are broken out into individual sub-channels/lanes 4 x 100 Gbps, increasing the graph degree d per router, which drastically lowers average hop counts and oversubscription ratios.
3. ShuffleBox Passive Hardware: To mitigate physical cabling complexity, RNG introduces a passive optical component called a ShuffleBox deployed in central "shuffle panels". Connections between routers inside a room or across rooms pass through ShuffleBoxes, which shuffle fiber pairs internally. This keeps physical cabling steps and cross-room trunk runs on par with structured topologies while preserving random logical graph connectivity.

Interconnect Component

• AWS RNG: ShuffleBox (Static, passive optical cable shuffle device)
• Google Jupiter: Apollo OCS (MEMS-based Optical Circuit Switch)

Topology Flexibility

• AWS RNG: Static/Fixed Graph: Random physical graph established on Day 1. Traffic engineering adapts routing over fixed paths.
• Google Jupiter: Reconfigurable/Dynamic: MEMS mirrors rotate to change logical port-to-port fiber mappings in real time.

Control Plane Execution

To understand this concept, let's break down the quote into its three core engineering components: Fully Distributed Link-State, 2 VRFs per Switch, and Deterministic Hashing of a Shared Key. Together, these mechanisms allow thousands of switches in an RNG network to make independent, local routing decisions that globally coordinate to form loop-free, multi-path routes—without needing a central SDN controller or non-commodity hardware.

Fully Distributed Link-State

Traditional Link-State (like OSPF/IS-IS): Every switch in the network periodically broadcasts its local connections (links) to every other switch. As a result, every switch maintains an identical copy of the global topology map.

How RNG uses it: Instead of calculating traditional shortest-path trees (like Dijkstra’s algorithm), each switch runs Spraypoint’s level-computation algorithm locally over this shared topology map to determine waypoint levels ($WP_0, WP_1, IR, OR$) for all destinations.

RNG only needs 2 VRFs Per Switch

What is a VRF? VRF stands for Virtual Routing and Forwarding.

Think of VRFs as the network equivalent of virtual machines for a router's routing table. A single physical switch or router running VRF can maintain multiple independent routing tables simultaneously. Packets arriving on an interface assigned to VRF A are looked up using VRF A's private routing table, completely isolated from VRF B. This allows network engineers to apply entirely different routing rules, next-hop choices, or isolation policies to different interfaces on the exact same switch ASIC.

Why Does RNG Need Only 2 VRFs Per Switch?

In previous expander graph proposals (like Spineless Data Centers), researchers attempted to route around shortest paths by using VRFs to create separate virtual layers for different non-shortest paths. However, that approach required a large number of VRFs. Commodity switch ASICs can only support a small number of VRFs, so those designs failed to scale.

RNG achieves Spraypoint routing using strictly 2 VRFs per switch regardless of network size:

1. VRF 1: The "Spraying" Table (Server-Facing Interfaces).

• Role: Handles incoming traffic originating directly from local servers in the rack.
• Rule: If a server sends a packet destined for another server in the datacenter, this VRF ignores destination shortest paths and sprays the traffic across all fabric-facing uplink interfaces using ECMP hashing.
• Exception: If the destination server is attached to the same local switch, it is delivered directly.

2. VRF 2: The "Pointing" Table (Fabric-Facing Interfaces).

• Role: Handles traffic arriving from neighboring switches over the fabric uplinks.
• Rule: Once a packet has been "sprayed" into the fabric, it enters VRF 2 on the receiving switch. This VRF follows Spraypoint's pointing rules (guiding packets through waypoints and concentric rings toward the destination t).

Why 2 VRFs are Sufficient? Loop Prevention.

The primary danger of spraying traffic to arbitrary neighboring switches is routing loops—a packet could be sprayed to a neighbor, which might immediately spray it back, causing an infinite ping-pong loop.
By isolating the operations into two distinct VRFs, RNG enforces a strict two-phase state transition:

Server -> Enters VRF 1 -> Spraying Phase (1 Hop) -> Enters VRF 2 -> Pointing Phase (Multi-Hop to Destination)

A packet from a server enters VRF 1, gets sprayed exactly once to a neighbor. When the neighboring switch receives the packet on its fabric uplink, the packet lands in VRF 2. From that moment on, the packet remains in VRF 2 across all subsequent transit hops, strictly obeying the waypoint pointing rules until it reaches the destination. Because VRF 2 uses deterministic, loop-free pointing graphs for every destination, a packet can revisit a source at most once during the initial spray step and will never loop indefinitely.

Therefore, using only 2 VRFs allows Spraypoint to scale to tens of thousands of switches while fitting comfortably within the hardware limits of off-the-shelf commodity switch ASICs.

• VRF 1 (Spraying): Applied only to server-facing ports. It takes traffic originating from local servers and sprays it across all available uplinks to neighboring switches using Equal-Cost Multipath (ECMP).
• VRF 2 (Pointing): Applied to fabric-facing uplink ports. Once a packet arrives over the fabric from another switch, it enters VRF 2 and strictly follows the Spraypoint pointing rules toward the destination.

Deterministic Hashing of a Shared Key

The Waypoint Selection Problem: Spraypoint requires that intermediate nodes forward traffic to p randomly selected waypoint neighbors surrounding a destination. If Switch A thinks WP_1(t) includes Switch X, but Switch B thinks WP_1(t) includes Switch Y, inconsistent paths and infinite forwarding loops can form.

The Solution: Every switch in the datacenter is configured with the same secret seed/key (e.g., Key = "AWS_RNG_2026").

How it works: When any switch needs to randomly select p waypoints for destination t, it calculates a hash function combining the shared key, the node ID, and

t = Hash(Shared Key + Node ID + Destination)

Because every switch runs the exact same hash function with the exact same key over the exact same topology map, all switches independently compute identical waypoint graphs for destination t without communicating with one another.

Example

• Switch S (Source): Connected to Server A.
• Switch A & Switch B: Neighboring switches connected to S.
• Switch T (Destination): Connected to Server B. T's neighbors are A and B.

      [Server A]
           |
       (Switch S) --- [VRF 1: Spray]
        /      \
       /        \
   (Switch A)  (Switch B)  <-- [VRF 2: Waypoint / Pointing]
       \        /
        \      /
       (Switch T)
           |
       [Server B]

Compared to Google Jupiter (SDN & OCS): Centralized SDN Controller: Compute cluster calculates global routing tables and pushes forwarding rules down to switches.