Qwen3-32B at 4 bits has a raw parameter payload of 16.4 GB. We want to know whether one DGX Spark with 128 GB of unified memory can serve it to a coding agent at a long context length. The estimated weights come in under the installed memory; the open question is the rest of the budget.
The rest of the budget has to cover:
- the KV cache that holds the state of active requests
- the runtime's working memory and overhead
- the operating system
- anything else running on the machine
We can compute part of the budget from the model card; other parts need runtime measurements. By the end we know which numbers the model card gives us, which only a measurement on the device can give, and which traffic paths to check before trusting a local setup.
We work in five steps:
- Identify the exact device and its installed memory.
- Separate inference on the Spark from the tool actions a coding agent runs.
- Calculate the raw parameter payload from the model card.
- Follow the memory a request adds on top of the weights.
- Check the API path and the traffic paths before trusting the setup.
A worked example at the end applies the steps to two long-context requests.
Identify the device before using its capacity
Before we plan memory, we pin down exactly which device we are planning for.
The hardware here is the NVIDIA DGX Spark, product identifier 940-54242-0000. NVIDIA's product page lists the GB10 Grace Blackwell Superchip and 128 GB LPDDR5x memory for that identifier, and its hardware overview describes the 128 GB unified system-memory configuration. We use NVIDIA's specifications for that configuration. The same product page also lists 64 GB configurations, but our memory plan assumes the 128 GB configuration throughout.
With this unified-memory architecture, where the CPU and GPU share the same installed memory, we need to check how much installed memory one serving process can use.
NVIDIA also specifies computing capacity, and the hardware overview lists up to 1 PFLOP at FP4 with sparsity. Generated tokens per second, time to first token and coding quality have to be measured independently of that specification.
Keep inference separate from the tool action
The memory budget covers the Spark alone, so we first need to be clear about which parts of a coding-agent session actually run on it.
A coding-agent harness prepares context and sends a request to a model API, and in this setup a server on the Spark runs inference and returns generated content to the harness. If the client and server need a translation adapter, we identify the API boundary where it sits and check its request and response behavior. When the response asks for a file operation, the harness or its tool coordinator decides whether to run the tool. The tool runtime, which might live on a workstation, inside a container, or elsewhere, performs the operation under its own permissions.
The model is software and has no filesystem access, and the tool runtime stays in its configured location when the Spark runs inference.
NVIDIA's one-device vLLM playbook covers setup on a DGX Spark or DGX Station. Its platform table distinguishes the two devices, and its advanced path includes API and tool-call checks. Docker and the Hugging Face CLI are prerequisites, while NVIDIA Sync and SSH requirements change with the path we choose. To follow the playbook, we need:
- a model revision
- a quantization (the bit width the weights are stored at)
- a named runtime version
- a context target
- a concurrency target

Figure text
The figure shows one interaction with a native or adapted request and response, device computation between them, and numbered directional arrows. The model response returns to the client. The agent then handles the tool call, its result, and the optional continuation. Either the agent or the tool runtime invokes the tool.
- The Spark answers, the tool runs elsewhere
- Dashed: optional part or step
- Lit Spark lamp: inference
- The Spark's memory budget covers inference only. Tools run in their own process under their own permissions, and the model has no file access.
- 1 Ask the model
- 2 Inference: Step 2 runs on the Spark
- 3 The answer comes back
- 4 Allowed action
- 5 Result of the action
- 6 Optional follow-up
- Coding agent: Prepares context, decides on tools
- API adapter: Only when configured. Maps both directions.
- DGX Spark: Model server: runs inference
- Tool executor: Invokes the permitted action
- Result for the model
- Tool runtime
- The agent can send that result to the model as a follow-up, over the same path.
Sources: [1] Serve LLMs with vLLM on One Device | DGX Spark (vllm); [2] Serve LLMs with vLLM on One Device | DGX Spark (instructions).
Calculate the raw parameter payload
With the components placed, we start the budget by calculating the weight size. Calculating the other budget values requires access to the hardware.
The Qwen3-32B model card gives a rounded parameter count of 32.8 billion, and with that count as P, the raw payload at b bits per parameter is P × b ÷ 8 bytes.
We divide bytes by 10^9 for decimal gigabytes or by 2^30 for gibibytes.
At 16 bits per parameter, 32.8 × 10^9 × 16 ÷ 8 gives 65.6 billion bytes, about 61.09 GiB.
Raw parameter payload by assumed bit width
We computed these from the model card’s rounded 32.8 billion count. Bytes = 32.8e9 × bits ÷ 8; GB = bytes ÷ 1e9; GiB = bytes ÷ 2^30. The script at the end of the next section prints the same three rows; we show three GiB decimals here.
Sources: qwen3 model card.
| Assumed bits / parameter | Raw payload (decimal GB) | Raw payload (GiB) |
|---|---|---|
| 16 | 65.6 | 61.095 |
| 8 | 32.8 | 30.547 |
| 4 | 16.4 | 15.274 |
Every row uses the same rounded parameter count and assumes every parameter takes exactly the stated width. Real quantization can:
- add metadata
- keep some tensors at other precisions
- need a particular packing format
We check that a matching checkpoint exists and that our runtime can run it.

Figure text
Three bars show the raw parameter payload calculated for 32.8 billion parameters at assumed 16, 8, and 4 bits per parameter. The separate runtime-budget worksheet covers weights, cache, runtime, system use, other processes, and reserve.
- Fewer bits, smaller payload
- Arithmetic from model card (bar)
- Measured only at runtime (blank)
- The three bars are arithmetic, assuming every number takes exactly that many bits. Check that a matching checkpoint exists, then measure the rest.
- Runtime budget, to measure: Weights; KV cache; Runtime overhead; Operating system and services; Other processes; Reserve
- 65.6 GB, 61.095 GiB
- 32.8 GB, 30.547 GiB
- 16.4 GB, 15.274 GiB
- 32.8 billion parameters
- 16 bits per parameter
- 8 bits per parameter
- 4 bits per parameter
- Runtime budget, to measure: Filled from measurements on the device
- Each number in 16 bits
- This is only the raw numbers. Six lines of the budget stay blank until measured on the machine.
Sources: [1] qwen3 model card.
Follow the memory used by a request
With the weights sized, we add the memory that each request brings with it.
Weights are the model data that inference needs. The runtime loads them in a supported format and allocates extra working memory to execute, and because those allocations vary with the model, kernels, and serving configuration, the raw-payload formula can't predict them.
The KV cache stores past key and value tensors that attention reuses during generation. Hugging Face's cache explanation describes how those tensors are reused instead of recomputed for each new token. With a growing full-context cache, every processed token adds stored state. How much more memory additional active requests take is set by the runtime's cache allocation and reuse policy. Other cache designs preallocate or keep only a sliding window, so context length alone doesn't give us the allocation. What the KV cache is and how much memory it takes estimates the bytes each cached position takes from a model's configuration file and lists the vLLM settings that size the cache.
Take a request that reads a long source file and then generates an edit. It uses the same weights as a short prompt and may need different amounts of cache and working memory. If we admit a second long request at the same time, the raw parameter payload estimate stays the same while the runtime holds two long requests.
Installed memory. NVIDIA lists a 128 GB configuration. We note which exact device we have and what its system report shows.
Raw parameter payload. From the arithmetic above. We note the actual checkpoint size in bytes and its format.
Fill the runtime budget worksheet using only your own runtime measurements.
Use one unit, take every measurement over the same time window, and keep the categories from overlapping. Record what each reported total includes; if it already includes weights or cache, replace those rows with that total to avoid double-counting. Pin the model revision, quantization, runtime, context, and concurrency.
| Budget component | How we account for it |
|---|---|
| Weights | We identify the actual checkpoint and quantization metadata, then measure the loaded weight allocation at runtime, excluding cache and overhead. |
| KV cache | We record context and concurrency and measure the cache allocation at runtime, excluding weights and overhead. |
| Runtime / allocator overhead | We identify the runtime and measure residual allocator use at runtime, excluding weights and KV cache already counted. |
| Operating system / services | We measure memory used by the OS and selected services at runtime, excluding the model runtime and the other processes listed separately; whole-system used memory belongs elsewhere. |
| Other processes | We measure concurrent process memory at runtime, excluding the runtime, OS and services counted above. |
| Unallocated reserve | We declare the headroom to leave unallocated and keep it separate from used memory and measured demand. |
We sum the nonoverlapping categories in one unit.
A short Python script holds both parts: the payload arithmetic from the model card, and the budget rows that stay empty until we measure them.
"""Raw parameter payload by bit width, and the budget rows to fill in."""
PARAMETERS = 32.8e9 # the Qwen3-32B model card's rounded parameter count
INSTALLED_GB = 128 # the DGX Spark configuration this post plans for
# Decimal GB measured on your own device (the reserve is the headroom you declare).
# None means not filled in yet.
BUDGET_GB = {
"Weights": None,
"KV cache": None,
"Runtime / allocator overhead": None,
"Operating system / services": None,
"Other processes": None,
"Unallocated reserve": None,
}
print("bits decimal GB GiB")
for bits in (16, 8, 4):
size = PARAMETERS * bits / 8 # bytes
print(f"{bits:>4} {size / 1e9:>10.1f} {size / 2**30:>7.3f}")
print(f"installed memory: {INSTALLED_GB} GB")
for name, gb in BUDGET_GB.items():
print(f" {name}: {'not filled in yet' if gb is None else f'{gb} GB'}")
if None not in BUDGET_GB.values():
print(f"sum of the rows: {sum(BUDGET_GB.values()):.1f} GB")
With nothing measured yet, it prints the payload table and lists the rows still to fill:
$ python3 payload_by_bits.py
bits decimal GB GiB
16 65.6 61.095
8 32.8 30.547
4 16.4 15.274
installed memory: 128 GB
Weights: not filled in yet
KV cache: not filled in yet
Runtime / allocator overhead: not filled in yet
Operating system / services: not filled in yet
Other processes: not filled in yet
Unallocated reserve: not filled in yet
For another model, change PARAMETERS to its parameter count.
Replace each None with the decimal GB you measure on your device, and the script adds the rows up.
Other checks for local inference
Once the budget fits, we confirm that the API path works for the agent and trace where every process sends its traffic.
The local API response shows how that path handled the request. One response does not cover every tool-call or streaming behavior a coding agent needs, so we also run a harmless tool call over the API path we chose and check that the harness finishes it.
The inference request may stay on the local network while model downloads, authentication, updates, or a tool action reach other destinations, so we check each process's traffic destinations separately from the model server's address. The model server and the model files it downloads are attack surfaces of their own, which Model servers are web apps and Model files are code cover. The table lists which component owns each path.
Traffic paths by process.
Sources: Serve LLMs with vLLM on One Device | DGX Spark (vllm) Serve LLMs with vLLM on One Device | DGX Spark (instructions).
| Traffic / execution class | How we trace it |
|---|---|
| Model and container downloads | We inspect the selected downloader's configuration and observe its process and destinations. |
| Authentication | We identify the client or service, its effective auth mechanism and its observed destination, without reading secret contents. |
| Telemetry | We read the current configuration and observe network traffic per process. |
| Updates | We read the updater's configuration and observe its owning process and destination. |
| Agent API request | We confirm the coding agent's exact process on the host, then check its effective endpoint and a request trace. Serve LLMs with vLLM on One Device | DGX Spark (vllm) Serve LLMs with vLLM on One Device | DGX Spark (instructions) |
| Agent tools | We note the allowed actions, identify the tool executor and observe where it sends traffic; destinations depend on the tool. |
Two long-context requests and the 4-bit estimate
We finish by applying the steps to one concrete target.
The 4-bit row gives 16.4 GB of raw parameter payload, and suppose we want Qwen3-32B to serve two long-context requests on the 128 GB Spark. To check whether it fits, we use a supported checkpoint and measure allocations at the target context and concurrency, including cache, execution overhead, and system demand.

- Installed memory: one DGX Spark with 128 GB of unified memory.
- Raw parameter payload: 32.8 billion parameters at 16 bits take 65.6 GB.
- One request: its KV cache and working memory sit on top of the weights, and only a runtime measurement gives their size.
- A second long request: the payload estimate stays the same while the runtime holds two long requests.
- The 4-bit estimate: the payload drops to 16.4 GB, and the rest of the budget is still measured on the device.
We also need to check coding quality on the same serving config to see how well it supports a coding agent.
The one thing to take away
The raw payload is arithmetic from the model card and installed memory comes from NVIDIA's specification; the remaining lines of the budget come from measuring the chosen serving configuration on the device itself, plus the reserve you declare.
What an attacker targets here
One Spark serving models runs two things an attacker wants: an HTTP server that manages models, and checkpoint files that can run code when loaded.
- A request to Ollama's
/api/pullmakes the server fetch and parse a manifest the caller chose, and CVE-2025-1975 turned a crafted one into a crash. We limit who may call pull and registration endpoints, as Model servers are web apps explains. - A pickle-based checkpoint can tell the loader to call any function.
We keep
weights_only=True, the default since PyTorch 2.6, and prefer a repository's safetensors files, as Model files are code shows.