Tensor parallelism splits every layer of a model across several devices, so those devices have to exchange results inside every layer. That one fact explains why the rest of this part spends so much time on cables, ranks and network checks. We define a tensor, split a small weight matrix with a script to show what the devices exchange, and list what vLLM's documentation and source say about the setting that controls the split.

What a tensor is

Every later term in this post is built on this one, so we take the definition from the library that most model code uses. PyTorch's documentation says: "A torch.Tensor is a multi-dimensional matrix containing elements of a single data type."

Two properties follow from that sentence:

Shape times element size gives the bytes a tensor occupies, which is the arithmetic behind every memory estimate in this part.

Sources: PyTorch 2.15 documentation, torch.Tensor and Tensor Attributes.

What a model's weights are

A model's weights are tensors too, and tensor parallelism is named after what it does to them. The Megatron-LM paper, whose algorithm vLLM's implementation includes, describes the unit that repeats: "A transformer layer consists of a self attention block followed by a two-layer, multi-layer perceptron (MLP)." The paper splits the matrix multiplies in both blocks, and vLLM's source defines the layer that performs one as Y = XA + b: an input X multiplied by a weight matrix A, plus a bias b.

In those terms, the weights that tensor parallelism splits are matrices like A, several in every layer. The configuration file of Qwen3-Coder-30B-A3B-Instruct, the model our KV cache post uses, declares 48 such layers.

Sources: Shoeybi et al., Megatron-LM, arXiv 1909.08053, section 3; vLLM v0.33.0 source, linear.py; Qwen, config.json at a pinned commit.

Three ways to use more than one device

Tensor parallelism is one of three strategies, and they differ in what each device holds. vLLM's documentation defines them this way:

The Megatron-LM paper states the limit of replicas alone: "the model must fit entirely on one worker." It describes pipeline parallelism as groups of operations "performed on one device before the outputs are passed to the next device in the pipeline", and calls its own method an "intra-layer model parallel approach." The paper, submitted in 2019, is about training; vLLM's documentation states that its inference implementation "includes Megatron-LM's tensor parallel algorithm."

Sources: vLLM v0.33.0 documentation, Optimization and Tuning, Data Parallel Deployment and Parallelism and Scaling; Shoeybi et al., Megatron-LM, abstract and section 2.3.

Splitting one weight matrix

A small example shows what holding a slice of a layer means and what the devices owe each other afterwards. This script is our illustration of the idea in plain Python, not vLLM's implementation. It multiplies one input by one 4 by 6 weight matrix, then repeats the work with the matrix cut in half two different ways, as if two workers each held one half:

"""Our illustration: one weight matrix split between two workers (not vLLM's code)."""


def apply(x, a):
    """Return y = x A for a row of inputs x and a weight matrix a."""
    return [sum(x[i] * a[i][j] for i in range(len(x))) for j in range(len(a[0]))]


x = [1, 2, 3, 4]  # one input: 4 numbers
a = [  # the weights: a 4 x 6 matrix, 4 inputs to 6 outputs
    [2, 0, 1, 3, 1, 0],
    [1, 1, 0, 2, 0, 4],
    [0, 3, 2, 1, 1, 1],
    [1, 0, 1, 0, 2, 3],
]
whole = apply(x, a)
print("shape of a:", (len(a), len(a[0])))
print("unsplit    :", whole)

# Split by columns: each worker keeps 3 of the 6 columns and receives the whole input.
left = [row[:3] for row in a]
right = [row[3:] for row in a]
y_left, y_right = apply(x, left), apply(x, right)
print("columns    :", y_left, "+", y_right, "placed side by side ->", y_left + y_right)
print("equal      :", y_left + y_right == whole)

# Split by rows: each worker keeps 2 of 4 rows and gets the matching half of the input.
top, bottom = a[:2], a[2:]
p_top, p_bottom = apply(x[:2], top), apply(x[2:], bottom)
total = [p + q for p, q in zip(p_top, p_bottom)]
print("rows       :", p_top, "and", p_bottom, "added ->", total)
print("equal      :", total == whole)

We ran it in a throwaway container with no network and a read-only filesystem:

$ docker run --rm --network none --read-only -v "$PWD":/demo:ro -w /demo python:3.12-slim python split_weights.py
shape of a: (4, 6)
unsplit    : [8, 11, 11, 10, 12, 23]
columns    : [8, 11, 11] + [10, 12, 23] placed side by side -> [8, 11, 11, 10, 12, 23]
equal      : True
rows       : [4, 2, 1, 7, 1, 8] and [4, 9, 10, 3, 11, 15] added -> [8, 11, 11, 10, 12, 23]
equal      : True

Both splits reproduce the unsplit result, and they differ in what the workers must do to get there:

vLLM's source has a layer type for each case. ColumnParallelLinear says the matrix "is parallelized along its second dimension as A = [A_1, ..., A_p]." RowParallelLinear says it "is parallelized along its first dimension and X along its second dimension", and when reduce_results is set and more than one device takes part, its forward method calls tensor_model_parallel_all_reduce on the output. That call is the addition in our last step, done across devices.

A panel shows one weight matrix of 4 rows by 6 columns, with its numbers. Two workers stand in front: Worker 1 holds rows 1 and 2, Worker 2 holds rows 3 and 4. Lanes numbered 1 carry a half of the table to each worker; lanes numbered 2 carry each worker's partial sum to a mark between them labelled Added. At the end the added result reads [8, 11, 11, 10, 12, 23].
  1. One weight matrix, two workers: the script multiplies one input by one 4 by 6 weight matrix, then repeats the work with the matrix cut in half two different ways, as if two workers each held one half.
  2. Split by columns: each worker computes a finished part of the output; the parts sit side by side and no arithmetic joins them.
  3. Split by rows: each worker computes a partial sum for every output.
  4. The two are added: neither half is the answer until the two are added.
The script is our illustration in plain Python, not the implementation in vLLM. In vLLM the row-parallel layer calls an all-reduce on its output when reduce_results is set and more than one device takes part; that call is this addition, done across devices.

Sources: our script, run with Python 3.12.16; vLLM v0.33.0 source, ColumnParallelLinear and RowParallelLinear.

Why the devices talk inside every layer

The addition in the row split cannot be skipped or postponed, and the Megatron-LM paper explains why. Between the two matrices of the MLP block sits a nonlinear function, GeLU, and a row split in front of it "will require a synchronization point before the GeLU function." The paper avoids that by ordering the two splits: "we partition the first GEMM in this column parallel fashion and split the second GEMM along its rows so it takes the output of the GeLU layer directly without requiring any communication." (GEMM is a matrix multiply.) The output of the second multiply "is then reduced across the GPUs."

The attention block gets the same treatment, split so that "the matrix multiply corresponding to each attention head is done locally on one GPU." The paper's total for its architecture is "only two all-reduces in the forward path and two in the backward path" per layer, and serving a model runs only the forward path.

vLLM's documentation states the consequence in one line: tensor parallelism "incurs significant communication overhead because of all-reduce being performed after every layer." A model with 48 layers repeats that exchange in all 48 for every token it generates.

Sources: Shoeybi et al., Megatron-LM, section 3; vLLM v0.33.0 documentation, Optimization and Tuning.

When the devices sit in different machines, every one of those exchanges crosses the network between them. vLLM's documentation is direct about it: "Efficient tensor parallelism requires fast internode communication, preferably through high-speed network adapters such as InfiniBand."

The same page describes GPUDirect RDMA as a technology "that allows network adapters to directly access GPU memory, bypassing the CPU and system memory", and gives a check for which path the runtime took. Start the server with NCCL_DEBUG=TRACE vllm serve ... and read the log:

Inside one machine the documentation makes a related call: for GPUs without an NVLINK interconnect, it recommends pipeline parallelism over tensor parallelism to lower the communication overhead. The documentation gives directions and no required link speed, so the link has to be measured on the hardware in hand. Planning a two-node DGX Spark TP2 setup does that for two machines joined by one cable.

Source: vLLM v0.33.0 documentation, Parallelism and Scaling.

Setting tensor_parallel_size in vLLM

One argument turns the split on. vLLM documents --tensor-parallel-size (short form -tp) as the "Number of tensor parallel groups", with a default of 1, and tells us to set it "to the desired GPU count."

The documentation's guide for a single model replica reads as three cases:

  1. If the model fits on a single GPU, "distributed inference is probably unnecessary."
  2. If it is too large for one GPU but fits on one node with several, "use tensor parallelism."
  3. If it is too large for one node, combine both: "Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes."

For a cluster, the page also allows a single tensor-parallel group across machines: "Alternatively, you can set tensor_parallel_size to the total number of GPUs in the cluster." That is the layout the DGX Spark posts in this part use, with one GPU per machine.

Two more documented requirements apply once machines are involved:

Sources: vLLM v0.33.0 documentation, Engine Arguments and Parallelism and Scaling.

Documented constraints on the split

Not every device count is a valid tensor-parallel size, and the rules live in two places.

The documentation covers uneven splits: "If the model fits within a single node but the GPU count doesn't evenly divide the model size, enable pipeline parallelism, which splits the model along layers and supports uneven splits."

The head-level rule is in source code, not in the documentation pages. In vLLM v0.33.0, the constructor of the Llama attention layer contains these lines, with three assertions:

assert self.total_num_heads % tp_size == 0
self.num_heads = self.total_num_heads // tp_size
self.total_num_kv_heads = num_kv_heads
if self.total_num_kv_heads >= tp_size:
    # Number of KV heads is greater than TP size, so we partition
    # the KV heads across multiple tensor parallel GPUs.
    assert self.total_num_kv_heads % tp_size == 0
else:
    # Number of KV heads is less than TP size, so we replicate
    # the KV heads across multiple tensor parallel GPUs.
    assert tp_size % self.total_num_kv_heads == 0
self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)

For that model class, the size has to divide the attention head count, which matches the paper's choice to keep each head on one GPU. This is one model file at one commit; other model classes may carry different checks, and a size that passes the arithmetic still has to load.

The same lines settle what happens to the heads whose keys and values the KV cache stores. While the tensor-parallel size is at most the KV head count, the KV heads are partitioned across the devices. Above that count they are replicated. Three places in the v0.33.0 source describe the replication: the comments above, the QKVParallelLinear docstring ("the key/value head may be replicated while the query heads are partitioned"), and the cache specification's max_tp_shards field, described as "Distinct shards this cache splits into across TP; TP ranks beyond this hold replicas." When eight-way tensor parallelism is worth testing applies the head arithmetic to one model's candidate sizes.

Sources: vLLM v0.33.0 documentation, Parallelism and Scaling; vLLM v0.33.0 source, llama.py, QKVParallelLinear and kv_cache_interface.py.

What tensor parallelism does not change

Tensor parallelism decides where the weights sit, and the documentation is specific about what it does around them.

Sources: vLLM v0.33.0 documentation, Optimization and Tuning, Parallelism and Scaling and Conserving Memory.

What an attacker targets here

The exchange inside every layer is network traffic, and vLLM documents that it is unprotected.

So the link that carries the split is part of the model server's attack surface, and it belongs on a network that nothing else can reach. Reach matters for the server's HTTP API too: every advisory in Model servers are web apps needed the attacker to reach an API, and it says the same rules hold for the vLLM servers in our DGX Spark posts.

Sources: vLLM v0.33.0 documentation, Parallelism and Scaling; vLLM v0.33.0 source, docs/usage/security.md.

Where to go next

Sharding the weights leaves each device more memory for the KV cache, and What the KV cache is and how much memory it takes works out how large that cache gets. For the memory budget of a single machine, read Planning memory for inference on one DGX Spark.