A model already serves on a four-device tensor-parallel group, and we want to know whether four more devices would make a coding agent feel faster.
If you run a model server for coding agents, this article shows how we decide whether eight-way tensor parallelism (TP8) deserves a test at all. The answer turns on which bottleneck we are hitting, and on whether the model's attention layout allows the larger split. We may need more cache for a bigger context or more independent serving capacity for many users, while one slow request may be communication-bound. The partition check below is pinned to a named model configuration and a specific vLLM commit, and the TP4-versus-TP8 comparison holds the workload fixed, so you can repeat the same steps for your own model and runtime.
By the end you will be able to:
- check whether a model's attention-head counts allow a candidate TP size
- name the resource you are short of and the parallel strategy that fits it
- set up a TP8 comparison that does not assume a speedup
A team filling an order gives the vocabulary we use throughout:
- workers sharing one operation are tensor parallelism
- sequential workstations are pipeline parallelism
- extra complete teams are replicas
Real model placement is less tidy, because tensors and layers vary in size and cooperating ranks, the devices that make up one group, have to exchange data. The model and runtime determine which partitions are supported and what they cost. Tensors and tensor parallelism shows the split on one small weight matrix and why the ranks exchange results inside every layer.
We work through the decision in six steps:
- Learn what the attention heads compute, because head counts set which splits are allowed.
- Apply one model's partition rule to the candidate TP sizes.
- Choose the parallel strategy that fits the resource we are short of.
- Map the logical group onto the physical fabric.
- Measure capacity at the context and concurrency we need.
- Compare the alternatives with matched measurements.
Why attention-head counts decide which splits are allowed
Whether a TP size is allowed at all depends on the model's attention-head counts.
Attention uses queries, keys, and values, written Q, K, and V. Attention compares a query with the keys to produce scores, and after scaling, masking where needed, and normalization, the resulting weights combine the values into an output. Vaswani and colleagues describe the mechanism in section 3.2 of Attention Is All You Need.
A small example, starting from already-computed attention weights, shows the two roles.
Two allowed positions get weights of 0.25 and 0.75, and one component of their value vectors is 4 and 8.
That output component is 0.25 × 4 + 0.75 × 8 = 7.
The keys helped set the weights, and the values supplied the content being combined.
An attention head runs a learned attention calculation over projected representations, and multiple heads let different projections contribute to the layer's result. The model architecture fixes the head count, while sequence length changes how much token state the computation handles.
In causal generation, a token can't attend to future tokens. The runtime can keep the keys and values it already computed for past positions and reuse them for the next position. That's the KV cache described in Hugging Face's cache explanation. The standard KV cache keeps past K and V tensors per layer for the next token's attention, which works without past queries.
Caching saves recomputing those past keys and values. Attention still runs over the relevant past context. A growing cache stores more token state as the sequence grows, while static and sliding-window caches allocate differently, so we include the runtime's cache policy with any context-capacity figure. What the KV cache is and how much memory it takes works out that stored state per position from the layer count, the KV head count and the head width.
Apply one model's partition rule
With the head counts in hand, we can check the arithmetic the runtime applies to a candidate TP size.
The Qwen3-32B configuration declares 64 attention heads, 8 key/value heads, and 64 layers. Its model card names the attention layout grouped-query attention (GQA). With different Q and KV counts, query heads share key/value heads.
At vLLM commit 2d9d3093ead234940fa7d93825171c880b065cca, the model-validation source rejects a TP size that doesn't evenly divide the total attention-head count.
The same file's head-count methods divide attention heads by TP size and return at least one KV head per GPU.
When there are fewer KV heads than TP ranks, that source describes KV-head replication so each GPU has at least one.
Of the three candidate TP sizes for Qwen3-32B, only those that divide the 64 query heads evenly pass this rule.
| Candidate TP size | Query-head division | KV heads per GPU under the cited rule | Static screening result |
|---|---|---|---|
| 3 | 64 ÷ 3 is not an integer | Not applicable: TP3 rejected | Fails the cited attention-head divisibility check |
| 4 | 64 ÷ 4 = 16 | 8 ÷ 4 = 2 | Passes this check |
| 8 | 64 ÷ 8 = 8 | 8 ÷ 8 = 1 | Passes this check |
A short Python script applies the same rule.
Its three fields are copied from the Qwen/Qwen3-32B configuration file at revision 9216db5781bf21249d130ec9da846c4624c16137:
"""Check candidate tensor-parallel sizes against a model's attention-head counts."""
import json
# Three fields of config.json from Qwen/Qwen3-32B at revision
# 9216db5781bf21249d130ec9da846c4624c16137.
CONFIG = json.loads("""
{
"num_attention_heads": 64,
"num_hidden_layers": 64,
"num_key_value_heads": 8
}
""")
TP_SIZES = (3, 4, 8)
heads, kv_heads = CONFIG["num_attention_heads"], CONFIG["num_key_value_heads"]
print(f"attention heads: {heads}, key/value heads: {kv_heads}, layers: {CONFIG['num_hidden_layers']}")
for tp in TP_SIZES:
if heads % tp:
print(f"TP{tp}: rejected, {heads} / {tp} is not an integer")
continue
# The cited rule returns at least one KV head per GPU.
per_gpu = max(1, kv_heads // tp)
print(f"TP{tp}: passes this check, {heads} / {tp} = {heads // tp} query heads per GPU, KV heads per GPU: {per_gpu}")
It prints one line per candidate size:
$ python3 tp_head_check.py
attention heads: 64, key/value heads: 8, layers: 64
TP3: rejected, 64 / 3 is not an integer
TP4: passes this check, 64 / 4 = 16 query heads per GPU, KV heads per GPU: 2
TP8: passes this check, 64 / 8 = 8 query heads per GPU, KV heads per GPU: 1
To screen another model, replace the three values with the ones in its config.json and set TP_SIZES to the sizes you are considering.

- The head counts: the Qwen3-32B configuration declares 64 attention heads and 8 key/value heads.
- TP3 is rejected: 64 ÷ 3 is not an integer, and the model-validation source rejects a TP size that does not evenly divide the attention-head count.
- TP4 passes this check: 64 ÷ 4 = 16 query heads and 8 ÷ 4 = 2 key/value heads per rank.
- TP8 passes this check: 64 ÷ 8 = 8 query heads and 8 ÷ 8 = 1 key/value head per rank; three more steps remain after this check.
We check the Qwen configuration against the pinned vLLM commit. After this check passes, three more steps remain:
- Load a checkpoint.
- Confirm that a kernel supports the hardware path.
- Complete requests.
The restriction comes from 64 heads and that specific validation code, so other models or runtimes need their own check.

Figure text
The partition diagram compares TP4, TP8, and TP3 arithmetic using Qwen3-32B head counts to narrow the candidates. It ends with runtime validation, where a pinned runtime and model load determine support.
- One check, then the real test
- Passes this one check (split size)
- A split size the check refuses
- Still to show on a real runtime
- The check narrows the candidates to four and eight. Neither has been loaded or run.
- Read the head counts
- Does not divide evenly; Divides evenly
- Qwen3-32B head counts
- Head-count check
- Result
- Refused
- Not crossed here
- Pinned runtime + model load
- 64 ÷ 3 is not whole
- Runtime validation
- 64 query heads; 8 key/value heads
- 4: 16 Q + 2 KV per rank
- 8: 8 Q + 1 KV per rank
- Both stop here. Whether either one loads, fits or is fast enough is a separate test on a real runtime.
Sources: [1] qwen3 config; [2] qwen3 model card; [3] pinned-vllm-model-rules; [4] Attention Is All You Need, first five pages for attention mechanism; [5] Transformers cache explanation.
Which parallel strategy fits the resource we are short of
Passing the head check tells us TP8 is allowed, and the decision to use it rests on which resource is short.
Tensor parallelism splits model computation and data across the ranks of one model instance, which can shrink some per-device allocations while communication and replicated state add costs. KV-head replication means that increasing TP can't always divide every cache component by the same factor.
Pipeline parallelism (PP) puts sequential model stages on different participants or groups, and a request's activations move from stage to stage. Different requests can occupy different stages, so we assess pipeline throughput and one request's latency separately. The pinned vLLM validation code checks that the model implementation supports PP before accepting a pipeline-parallel configuration.
Replicas are separate serving instances for independent requests. For a dense model, two complete TP4 replicas each hold a full instance of the model spread across four devices. Only the selected replica's group handles a request in this eight-device setup, and the request router and scheduling policy decide how the replicas get used.
Mixture-of-experts serving uses a different arrangement. vLLM's scaling documentation describes combining data-parallel attention with expert- or tensor-parallel expert layers. Its data-parallel deployment documentation explains that these groups may depend on one another through synchronization. Expert parallelism specifies where expert computation runs, while ordinary TP and two independent complete copies are two other arrangements.
Logical groups and the physical fabric
Whichever grouping we choose has to run over the cabling we actually have.
Take eight devices labeled A through H. One arrangement puts all eight in a single TP8 group, and another puts A through D in one TP4 replica and E through H in a second. These are two hypothetical groupings of the same eight Spark devices.
Both could run over one shared, compatible physical network. A shared fabric can still have contention, and its traffic and isolation follow from how the fabric is configured and used. The grouping tells the runtime which participants cooperate, and the physical network provides the paths between devices.
NVIDIA's cabling guidance separates helper-supported switched clusters of up to four devices from larger clusters that need manual configuration or your own automation. Following that cabling guidance is one step, and an eight-Spark TP8 serving recipe still needs its own validation.

Figure text
One hypothetical dense-model diagram shows the same eight Sparks grouped two ways, with one rank per device on a shared fabric. One grouping is TP8 with PP1 and one complete replica; the other is TP4 with PP1 and two complete replicas. Each request uses its selected replica. Two TP4 groups can share one compatible fabric.
- Same eight machines, two groupings
- Solid line: a cable, never moves
- Frame: a group, a software choice
- Lit lamp: machine on the request
- Two hypothetical groupings of the same eight machines on the same cables. Which one helps is decided by a matched measurement, not by this picture.
- Request 1
- A request to copy A; A request to copy B
- Shared physical fabric
- Spark A
- Spark B
- Spark C
- Spark D
- Spark E
- Spark F
- Spark G
- Spark H
- Request router
- One group of eight
- Request 2
- The other copy can take a different request. A router decides which copy gets which.
Sources: [1] Personal AI Supercomputer Powered by Blackwell | NVIDIA DGX Spark; [2] Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (connect multiple sparks); [3] Connect Multiple DGX Sparks for Distributed Workloads | DGX Spark (cable devices).
Whether the grouping holds our context and concurrency
Once a grouping is supported and cabled, the next question is whether it holds the context and concurrency we need.
We measure the runtime's overhead alongside weights and KV cache, since all three contribute to the serving allocation. Context length and the number of simultaneous requests set what that allocation has to hold. The total memory across machines remains noncontiguous, and available memory varies by rank, so the runtime has to place each component on a device that can hold it.
vLLM's scaling guide describes log fields for KV-cache token capacity and estimated concurrency at a given request length. We measure user latency at that load separately from the concurrency estimate.
The Qwen model card states native context up to 32,768 tokens and describes YaRN validation up to 131,072 tokens.
It explains the default max_position_embeddings of 40,960 as 32,768 output tokens plus 8,192 tokens for typical prompts.
We check whether a given Spark runtime has enough KV-cache room for that context at the concurrency we want.
Compare the alternatives
With the constraint named, we can pick the candidate worth testing and the measurement that decides it.
Required model or request state does not fit per device. Candidate: More TP ranks or supported PP placement. Decided by: Per-device allocation, cache capacity, and successful requests.
One instance fits, but independent requests queue. Candidate: Additional replicas or changed concurrency settings. Decided by: Queue delay, aggregate throughput, and per-request latency.
One request is slow across a distributed group. Candidate: Different partition or communication arrangement. Decided by: Prefill (prompt processing) and decode (token generation) timings and communication readings.
A proposed TP size fails the model rule. Candidate: Compatible TP size, supported PP, or another model/runtime. Decided by: Versioned constraint check and an actual load/inference result.
Faster output fails coding tasks. Candidate: Model or serving configuration changes. Decided by: Coding-quality evaluation paired with throughput.
Use the same workload, model, quantization, runtime, context, concurrency and quality method when comparing strategies. Report hardware specifications, collectives (the group communication operations between ranks), application performance and quality separately.
| Decision question | Measurement method |
|---|---|
| Model weight fit | We measure weights for the pinned checkpoint, quantization and runtime. |
| Runtime overhead | We measure the runtime's overhead alongside weights and KV cache, since all three contribute to the serving allocation. |
| KV cache / context / concurrency | We measure cache allocation and capacity at the target context and concurrency. |
| Latency / throughput | We match the workload, request distribution and serving conditions. |
| Coding-task quality | We match the task set, scoring method and model configuration. |
| Network / collective cost | We record observations of the fabric, transport and collectives under separate labels. |
| Utilization / operations | We record request allocation, utilization and observations from operating the system. |
Sources: nccl tests vllm parallel config Parallelism and Scaling - vLLM Hardware Overview - DGX Spark User Guide.
For a TP4-versus-TP8 comparison, we hold these fixed:
- model revision
- quantization
- runtime revision
- prompt/output workload
- measurement method
We state the request arrival pattern and target concurrency, so a two-replica result can be compared with one larger group. We write down every change that comes from the parallelism itself. To assess long-context capacity, we test at that context length.
The measurement terms need precise meanings before the numbers can be compared. Time to first token is the elapsed time from our chosen request-start event to the first output token, either emitted by the server or received by the client, and we state which one we measure. Decode throughput is generated tokens per unit time, reported separately per request and in aggregate. Link rate, RDMA bandwidth, and collective bandwidth describe different work in different units.
How the steps combine for a TP4 instance with users waiting
A TP4 instance fits the model and target context, but independent users are waiting in its queue. The head arithmetic also allows TP8. We could compare one TP8 group with two TP4 replicas.
What to do next
Eight devices are worth testing when a named constraint points to them and the model's partition rule allows them, and a matched measurement decides whether they help. Start by running the partition check for your own model and runtime, since the 64-head result above holds only for Qwen3-32B at the pinned vLLM commit. Every node loads the same weights, so pin their revision and use a format that stores only tensors, as Model files are code explains.
What an attacker targets here
Eight devices load every file in the model repository, and each file is a place to hide a payload.
- The EvilModel paper hid 36.9 MB of malware in a 178 MB AlexNet model with under 1% accuracy loss, and VirusTotal engines raised no suspicion. Its payload needs a receiver program to run, so the control is knowing which file we expected and checking that we got it, as Model files are code explains.
- MITRE ATLAS lists chat templates, tokenizer metadata, configuration and processing logic as parts of a model package that can carry unsafe behavior. Those files need reading before any device loads them, not only the tensors, as the section on safetensors explains.