A triangle of three DGX Sparks is one question and a four-device TP4 group is another, and the cabling drawing alone does not answer the second.
This article is for anyone with three Sparks cabled in a triangle who wants to serve a model with four-way tensor parallelism (TP4, which splits one model across four devices). The drawing shows physical links among three machines and does not tell you whether these Sparks form a tensor-parallel group or could run TP4, so we check the four-rank group's requirements alongside the cabling. By the end you can read a drawing as a complete link set, place four ranks on four CUDA devices, check the model partition, and know which switch, port and model details to confirm before you buy a cable or start a run. The sources are NVIDIA's playbook and clustering documents, the NCCL user guide, the Qwen3-32B configuration, the vLLM source at a pinned commit, and the MikroTik datasheet, all linked below.
Two definitions carry through the article. Physical links show the connections between participants, and a communication plan shows how they move data through those links. We use “physical loop” to describe cabling and “ring” to name a collective algorithm, the pattern a group communication such as AllReduce follows.
We check a candidate in seven steps, one per section:
- Draw every link in the three-node example and compare it with the complete link set for each layout.
- Place four ranks on four CUDA devices and check that the model splits across them.
- Get the collective route from the runtime version, transport configuration and logs.
- Count switch ports and Spark endpoints.
- Separate management access from the data fabric.
- Run the compatibility checks.
- Count compute devices: the switch lacks a CUDA device for a rank.
Draw every link in the three-node example
The first step turns the drawing into an explicit list of links.
Call the machines Spark A, Spark B, and Spark C. The direct triangle has exactly three bidirectional links: A-B, B-C, and C-A. Because every machine connects directly to every other, the topology at three nodes is both a ring and a full mesh.
At four nodes, a ring and a full mesh have different link sets. A four-node ring of A-B, B-C, C-D, and D-A has no direct A-C or B-D cable, so a full mesh would require more connections than the drawing shows.
A short Python script lists both link sets for three and four nodes:
"""Links in a direct ring and in a full mesh of the same nodes."""
from itertools import combinations
NODE_COUNTS = (3, 4) # the two direct layouts this post compares
for n in NODE_COUNTS:
nodes = "ABCDEFGH"[:n]
ring = [(nodes[i], nodes[(i + 1) % n]) for i in range(n)]
mesh = list(combinations(nodes, 2)) # every pair: n * (n - 1) / 2 links
in_ring = {frozenset(link) for link in ring}
extra = [link for link in mesh if frozenset(link) not in in_ring]
print(f"{n} nodes: ring {len(ring)} links, full mesh {len(mesh)} links")
print(f" ring: {', '.join(f'{a}-{b}' for a, b in ring)}")
print(f" mesh links the ring lacks: {', '.join(f'{a}-{b}' for a, b in extra) or 'none'}")
print(f" links at each node: ring 2, full mesh {n - 1}")
At three nodes the two sets are the same, and at four the ring lacks the two diagonal links:
$ python3 ring_and_mesh.py
3 nodes: ring 3 links, full mesh 3 links
ring: A-B, B-C, C-A
mesh links the ring lacks: none
links at each node: ring 2, full mesh 2
4 nodes: ring 4 links, full mesh 6 links
ring: A-B, B-C, C-D, D-A
mesh links the ring lacks: A-C, B-D
links at each node: ring 2, full mesh 3
To check a larger group, add its node count to NODE_COUNTS.
NVIDIA's cabling playbook documents these layouts:
- a two-device direct link
- a three-device direct ring
- a four-device direct ring
- switched layouts
In its switched setup, each Spark connects to its own switch port, and the playbook warns against mixing direct device links with switched links there.
Compare the cables in your drawing with the complete link set for each layout below.
| Physical example | Complete link set | What the drawing shows |
|---|---|---|
| Two-node direct | A-B | One device-to-device connection |
| Three-node direct triangle | A-B, B-C, C-A | Ring and full mesh for exactly three nodes |
| Three-node switched star | A-S, B-S, C-S, where S is the switch | Each node connects to the switch |
| Four-node direct ring, Configure Manually only | A-B, B-C, C-D, D-A | A closed loop with no diagonal links |
The playbook limits the four-device direct ring to the Configure Manually tab. NVIDIA Sync supports up to three directly cabled Sparks and up to four through a switch. The three-node switched layout replaces the triangle's direct links with links to the switch, S.

Figure text
The figure goes through four layouts in turn: a two-node cable, all three bidirectional links in a three-node triangle that forms both a direct ring and a full mesh, a complete four-device ring, and four switch-to-Spark cables. The picture shows the last of them.
- Four layouts, counted cable by cable
- Solid line: a plugged-in cable
- Dashed outline: no cable here
- A layout is a list of cables. Ranks and the communication plan are checked separately.
- Switch, S
- Spark D
- Spark C
- Spark A
- Spark B
- These are cables only. How the machines share work is a separate check.
Sources: [1] ConnectX-7 Networking - DGX Spark User Guide; [2] nccl tests; [3] Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices).
Place four ranks on four devices and check the model partition
A TP4 group needs four CUDA devices, so the second step takes four machines and a model that splits four ways.
For the TP4 candidate, we take four devices on four machines: Spark A, Spark B, Spark C, and Spark D. We put one rank on each CUDA device, as the table shows. These four placements make a separate group from the three-node layout. Each rank then holds a slice of every layer's weights, as Tensors and tensor parallelism explains, so all four take part in every layer.
| TP4 rank | Example device placement |
|---|---|
| 0 | CUDA device on Spark A |
| 1 | CUDA device on Spark B |
| 2 | CUDA device on Spark C |
| 3 | CUDA device on Spark D |
The model also has to split evenly across the four ranks.
Loading the model on Spark is what shows whether the partition works.
The Qwen3-32B configuration declares 64 attention heads and 8 key/value heads.
At vLLM commit 2d9d3093ead234940fa7d93825171c880b065cca, the model-validation code checks that the total attention-head count divides evenly by TP size.
64 ÷ 4 = 16 passes.
64 ÷ 3 is not an integer.
Check separately whether rank counts must be powers of two in general and whether the candidate loads on Spark. The same source checks whether a model implementation supports pipeline parallelism when PP is requested, so confirm implementation support for each parallel strategy.
Collective route selection happens outside the switch
Rank placement and cabling alone do not determine how the ranks exchange data.
NCCL defines AllReduce as a reduction among ranks that returns the result to each rank, and the definition leaves the physical route unspecified. The serving runtime and communication library decide how data moves over the configured fabric. So a switched topology can carry communication organized as a logical ring.
"Ringless" describes devices cabled without a closed physical loop. To know the actual collective route, we need the runtime version, the transport configuration, and the runtime's own logs.
Switch port counts and Spark endpoints
If the TP4 candidate connects through a switch, the port counts on the Spark side and the switch side decide how many devices the fabric can connect.
NVIDIA's ConnectX-7 guide describes two external QSFP ports per Spark, each up to 200 Gb/s, and explains that the port-to-PCIe layout produces multiple software interfaces. We need details beyond the physical port label to assign every runtime network interface or RDMA adapter.
MikroTik's CRS804-4DDQ datasheet lists four 400 Gb/s QSFP-DD ports and breakout options including two 200 Gb/s links per port.
4 × 2 = 8 counts the possible endpoint links in that arrangement.
NVIDIA's switched cabling procedure calls for three things:
- a supported QSFP112 DAC (a direct-attach cable) in Ethernet mode
- one cable from each Spark to its own 200 Gb/s Ethernet switch port
- the same Spark port number on every device, either port 0 or port 1
A 400 Gb/s switch port may need splitting into two 200 Gb/s ports. NVIDIA notes that changing a high-speed port's configuration can restart other links in the same port group, so we plan the split before the cluster carries traffic.
Before running eight Sparks at once or choosing a cable or switch config, confirm which of the following work with the Spark ports:
- CRS804 breakout
- cable
- port mode
- firmware

Figure text
The diagram shows Spark QSFP ports separately from four CRS804 ports.
- Count ports, then confirm they fit
- Lit port lamp: vendor lists it
- Eight possible links on paper. Each one still needs a compatible cable, port mode and firmware. Nothing here has been connected.
- 4 × 2 = 8 possible links
- Spark A: Two QSFP ports, each up to 200 Gb/s
- CRS804 switch: Breakout: two 200G links per port
- Confirm before cabling: Which of these work with the Spark ports
- Possible endpoint links
- QSFP port 0
- Port 1: 400G
- CRS804 breakout; Cable; Port mode; Firmware
- Cable, not plugged in
- Confirm these four things with the Spark ports before choosing a cable.
Sources: [1] ConnectX-7 Networking - DGX Spark User Guide; [2] crs804 datasheet; [3] CRS804 DDQ; [4] Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices).
Separate management access from the data fabric
The cluster also needs management access, which NVIDIA keeps on a separate path from the data links above.
NVIDIA's clustering overview uses a common LAN for management access and configuration, while the high-speed ConnectX-7 fabric carries traffic between devices once the right interfaces and network are configured. Configuring the fabric is a separate role from participating as a model rank, even for a component with access to every machine. A model server's management API belongs on the management side and off any network an attacker can reach, as Model servers are web apps explains.
For the switched path, the cabling instructions give three settings:
- put the Spark-facing ports on a common Layer 2 network
- keep the default MTU
- check hardware forwarding where the switch reports it
Check whether a TP4 candidate needs PFC (priority flow control), ECN (explicit congestion notification), or other network tuning.
According to the playbook, the NVIDIA Sync app and the manual setup helper each support up to four devices through a switch, and it separately discusses manual configuration or your own automation for larger switched clusters. Helper and model-server support each need checking independently of the switch's theoretical connection count.
Compatibility checks
Each question about the candidate has evidence that answers it.
| Condition | Documented requirements |
|---|---|
| Port speed / breakout mode | The datasheet lists four 400G ports and describes two 200G links per port. crs804 datasheet CRS804 DDQ |
| Cable / transceiver | The NVIDIA procedure calls for a supported QSFP112 DAC (a direct-attach cable) in Ethernet mode. In its Configure Manually helper, a direct pair uses matching Spark port numbers. Separately, the switched procedure uses the same port number on every Spark. Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices) |
| Layout / configuration path | NVIDIA documents the four-device direct ring under Configure Manually only. Sync supports up to three direct or four switched Sparks. ConnectX-7 Networking - DGX Spark User Guide Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices) |
| Layer 2 / MTU | The NVIDIA switched procedure requires a common Layer 2 network and the default MTU, and does not allow mixing direct and switched links. Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices) |
| Model / TP / PP support | vLLM 2d9d3093ead234940fa7d93825171c880b065cca checks head divisibility and model PP support. pinned-vllm-model-rules qwen3 config |
Sources: crs804 datasheet CRS804 DDQ Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices) Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (troubleshooting) pinned-vllm-model-rules ConnectX-7 Networking - DGX Spark User Guide qwen3 config.

Figure text
The selector offers separate branches for physical, network, RDMA, collective and serving observations, each with its own measurements. A serving test shows what the application does, separately from any RDMA, NCCL or microbenchmark result.
- Pick the test for the question
- Lit lamp: test picked for question
- Card block: outcome and its scope
- A result answers only the question its test was built for. No test on this page has been run for you.
- 1 Bring one question
- 2 Which cables go where?; Can the machines reach each other?; Does the direct transfer run?; Does the group operation run?; What does the model server answer?
- 3 Each result goes on the record
- Write down the outcome: Whether it ran, what happened; What it covered; The versions you used
- Your question
- Choose a branch
- Physical layout
- Network path
- RDMA question
- Collective question
- Serving the chosen model
- Outcome + what it covered
- The model's response; The serving configuration; Agent: request + tool result
- Two questions, two outcomes, kept apart. One does not stand in for the other.
Sources: [1] Serve LLMs with vLLM on One Device | DGX Spark (vllm); [2] nccl tests; [3] Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices); [4] Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (troubleshooting).
A switch cannot serve as the fourth rank
The last check counts compute devices, because a switch in the star forwards traffic without adding a fourth rank.
In a three-Spark star, A, B, and C connect to a switch. Using the three Sparks and the switch as ranks 0 through 3 would put a rank on the switch in the four-device TP4 candidate.
The switch forwards traffic and lacks a CUDA device for a rank, so the star has three compute devices where the TP4 candidate needs four.

- The direct triangle: exactly three bidirectional links, A-B, B-C and C-A.
- The switched star: links to the switch, S, replace the triangle's direct links.
- Ranks 0 through 3: using the three Sparks and the switch as the four ranks would put a rank on the switch.
- The switch is not a compute device: it forwards traffic and lacks a CUDA device for a rank, so the star has three compute devices where the TP4 candidate needs four.
Next step
Work through the compatibility table for your own cluster before you buy a cable or start a run.
What an attacker targets here
Three nodes, a switch and a separate management network add places to tamper with weights and to reach a listener nobody meant to open.
- Safetensors does not check tensor values, and PoisonGPT changed a model's parameters directly and was published to Hugging Face. We record the hash of every file each node loads, as Model files are code recommends, and compare them across nodes.
- The official Ollama image listens on every interface, and in our check listing models answered HTTP 200 with no credential.
We read each node's listeners with
ss -ltrbefore it joins the fabric, the check in Model servers are web apps.