Qwen3-32B at 4 bits has a raw parameter payload of 16.4 GB. We want to know whether one DGX Spark with 128 GB of unified memory can serve it to a coding agent at a long context length. The estimated weights come in under the installed memory; the open question is the rest of the budget.

The rest of the budget has to cover:

We can compute part of the budget from the model card; other parts need runtime measurements. By the end we know which numbers the model card gives us, which only a measurement on the device can give, and which traffic paths to check before trusting a local setup.

We work in five steps:

  1. Identify the exact device and its installed memory.
  2. Separate inference on the Spark from the tool actions a coding agent runs.
  3. Calculate the raw parameter payload from the model card.
  4. Follow the memory a request adds on top of the weights.
  5. Check the API path and the traffic paths before trusting the setup.

A worked example at the end applies the steps to two long-context requests.

Identify the device before using its capacity

Before we plan memory, we pin down exactly which device we are planning for.

The hardware here is the NVIDIA DGX Spark, product identifier 940-54242-0000. NVIDIA's product page lists the GB10 Grace Blackwell Superchip and 128 GB LPDDR5x memory for that identifier, and its hardware overview describes the 128 GB unified system-memory configuration. We use NVIDIA's specifications for that configuration. The same product page also lists 64 GB configurations, but our memory plan assumes the 128 GB configuration throughout.

With this unified-memory architecture, where the CPU and GPU share the same installed memory, we need to check how much installed memory one serving process can use.

NVIDIA also specifies computing capacity, and the hardware overview lists up to 1 PFLOP at FP4 with sparsity. Generated tokens per second, time to first token and coding quality have to be measured independently of that specification.

Keep inference separate from the tool action

The memory budget covers the Spark alone, so we first need to be clear about which parts of a coding-agent session actually run on it.

A coding-agent harness prepares context and sends a request to a model API, and in this setup a server on the Spark runs inference and returns generated content to the harness. If the client and server need a translation adapter, we identify the API boundary where it sits and check its request and response behavior. When the response asks for a file operation, the harness or its tool coordinator decides whether to run the tool. The tool runtime, which might live on a workstation, inside a container, or elsewhere, performs the operation under its own permissions.

The model is software and has no filesystem access, and the tool runtime stays in its configured location when the Spark runs inference.

NVIDIA's one-device vLLM playbook covers setup on a DGX Spark or DGX Station. Its platform table distinguishes the two devices, and its advanced path includes API and tool-call checks. Docker and the Hugging Face CLI are prerequisites, while NVIDIA Sync and SSH requirements change with the path we choose. To follow the playbook, we need:

A coding agent and a DGX Spark, with a dashed outline between them for an API adapter, marked only when configured. A separate frame near the agent holds a tool executor. Lanes: 1 ask the model, 3 the answer comes back, 4 allowed action, 5 result of the action, and a dashed lane 6, optional follow-up. A floor mark numbered 2, Inference, stands by the Spark. At the end a card tagged Result for the model rests at the Spark.
The coding agent receives inference output, coordinates an allowed tool action, receives its result, and can send a continuation request. If we use an adapter, it must handle both directions. The figure shows one interaction in numbered steps, with step 2 between request 1 and response 3.
Figure text

The figure shows one interaction with a native or adapted request and response, device computation between them, and numbered directional arrows. The model response returns to the client. The agent then handles the tool call, its result, and the optional continuation. Either the agent or the tool runtime invokes the tool.

  • The Spark answers, the tool runs elsewhere
  • Dashed: optional part or step
  • Lit Spark lamp: inference
  • The Spark's memory budget covers inference only. Tools run in their own process under their own permissions, and the model has no file access.
  • 1 Ask the model
  • 2 Inference: Step 2 runs on the Spark
  • 3 The answer comes back
  • 4 Allowed action
  • 5 Result of the action
  • 6 Optional follow-up
  • Coding agent: Prepares context, decides on tools
  • API adapter: Only when configured. Maps both directions.
  • DGX Spark: Model server: runs inference
  • Tool executor: Invokes the permitted action
  • Result for the model
  • Tool runtime
  • The agent can send that result to the model as a follow-up, over the same path.

Sources: [1] Serve LLMs with vLLM on One Device | DGX Spark (vllm); [2] Serve LLMs with vLLM on One Device | DGX Spark (instructions).

Calculate the raw parameter payload

With the components placed, we start the budget by calculating the weight size. Calculating the other budget values requires access to the hardware.

The Qwen3-32B model card gives a rounded parameter count of 32.8 billion, and with that count as P, the raw payload at b bits per parameter is P × b ÷ 8 bytes. We divide bytes by 10^9 for decimal gigabytes or by 2^30 for gibibytes. At 16 bits per parameter, 32.8 × 10^9 × 16 ÷ 8 gives 65.6 billion bytes, about 61.09 GiB.

Raw parameter payload by assumed bit width

We computed these from the model card’s rounded 32.8 billion count. Bytes = 32.8e9 × bits ÷ 8; GB = bytes ÷ 1e9; GiB = bytes ÷ 2^30. The script at the end of the next section prints the same three rows; we show three GiB decimals here.

Sources: qwen3 model card.

Assumed bits / parameterRaw payload (decimal GB)Raw payload (GiB)
1665.661.095
832.830.547
416.415.274

Every row uses the same rounded parameter count and assumes every parameter takes exactly the stated width. Real quantization can:

We check that a matching checkpoint exists and that our runtime can run it.

Three bars on one scale: 16 bits per parameter, 65.6 GB (61.095 GiB); 8 bits per parameter, 32.8 GB (30.547 GiB); 4 bits per parameter, 16.4 GB (15.274 GiB). A disc names the count, 32.8 billion parameters, with a lane to each bar. A panel named Runtime budget, to measure lists six lines: weights, KV cache, runtime overhead, operating system and services, other processes, reserve.
The bars plot the table's three raw-payload values, calculated from the model card's rounded 32.8B count.
Figure text

Three bars show the raw parameter payload calculated for 32.8 billion parameters at assumed 16, 8, and 4 bits per parameter. The separate runtime-budget worksheet covers weights, cache, runtime, system use, other processes, and reserve.

  • Fewer bits, smaller payload
  • Arithmetic from model card (bar)
  • Measured only at runtime (blank)
  • The three bars are arithmetic, assuming every number takes exactly that many bits. Check that a matching checkpoint exists, then measure the rest.
  • Runtime budget, to measure: Weights; KV cache; Runtime overhead; Operating system and services; Other processes; Reserve
  • 65.6 GB, 61.095 GiB
  • 32.8 GB, 30.547 GiB
  • 16.4 GB, 15.274 GiB
  • 32.8 billion parameters
  • 16 bits per parameter
  • 8 bits per parameter
  • 4 bits per parameter
  • Runtime budget, to measure: Filled from measurements on the device
  • Each number in 16 bits
  • This is only the raw numbers. Six lines of the budget stay blank until measured on the machine.

Sources: [1] qwen3 model card.

Follow the memory used by a request

With the weights sized, we add the memory that each request brings with it.

Weights are the model data that inference needs. The runtime loads them in a supported format and allocates extra working memory to execute, and because those allocations vary with the model, kernels, and serving configuration, the raw-payload formula can't predict them.

The KV cache stores past key and value tensors that attention reuses during generation. Hugging Face's cache explanation describes how those tensors are reused instead of recomputed for each new token. With a growing full-context cache, every processed token adds stored state. How much more memory additional active requests take is set by the runtime's cache allocation and reuse policy. Other cache designs preallocate or keep only a sliding window, so context length alone doesn't give us the allocation. What the KV cache is and how much memory it takes estimates the bytes each cached position takes from a model's configuration file and lists the vLLM settings that size the cache.

Take a request that reads a long source file and then generates an edit. It uses the same weights as a short prompt and may need different amounts of cache and working memory. If we admit a second long request at the same time, the raw parameter payload estimate stays the same while the runtime holds two long requests.

Installed memory. NVIDIA lists a 128 GB configuration. We note which exact device we have and what its system report shows.

Raw parameter payload. From the arithmetic above. We note the actual checkpoint size in bytes and its format.

Fill the runtime budget worksheet using only your own runtime measurements.

Use one unit, take every measurement over the same time window, and keep the categories from overlapping. Record what each reported total includes; if it already includes weights or cache, replace those rows with that total to avoid double-counting. Pin the model revision, quantization, runtime, context, and concurrency.

Budget componentHow we account for it
WeightsWe identify the actual checkpoint and quantization metadata, then measure the loaded weight allocation at runtime, excluding cache and overhead.
KV cacheWe record context and concurrency and measure the cache allocation at runtime, excluding weights and overhead.
Runtime / allocator overheadWe identify the runtime and measure residual allocator use at runtime, excluding weights and KV cache already counted.
Operating system / servicesWe measure memory used by the OS and selected services at runtime, excluding the model runtime and the other processes listed separately; whole-system used memory belongs elsewhere.
Other processesWe measure concurrent process memory at runtime, excluding the runtime, OS and services counted above.
Unallocated reserveWe declare the headroom to leave unallocated and keep it separate from used memory and measured demand.

We sum the nonoverlapping categories in one unit.

A short Python script holds both parts: the payload arithmetic from the model card, and the budget rows that stay empty until we measure them.

"""Raw parameter payload by bit width, and the budget rows to fill in."""
PARAMETERS = 32.8e9  # the Qwen3-32B model card's rounded parameter count
INSTALLED_GB = 128  # the DGX Spark configuration this post plans for
# Decimal GB measured on your own device (the reserve is the headroom you declare).
# None means not filled in yet.
BUDGET_GB = {
    "Weights": None,
    "KV cache": None,
    "Runtime / allocator overhead": None,
    "Operating system / services": None,
    "Other processes": None,
    "Unallocated reserve": None,
}

print("bits  decimal GB      GiB")
for bits in (16, 8, 4):
    size = PARAMETERS * bits / 8  # bytes
    print(f"{bits:>4}  {size / 1e9:>10.1f}  {size / 2**30:>7.3f}")

print(f"installed memory: {INSTALLED_GB} GB")
for name, gb in BUDGET_GB.items():
    print(f"  {name}: {'not filled in yet' if gb is None else f'{gb} GB'}")
if None not in BUDGET_GB.values():
    print(f"sum of the rows: {sum(BUDGET_GB.values()):.1f} GB")

With nothing measured yet, it prints the payload table and lists the rows still to fill:

$ python3 payload_by_bits.py
bits  decimal GB      GiB
  16        65.6   61.095
   8        32.8   30.547
   4        16.4   15.274
installed memory: 128 GB
  Weights: not filled in yet
  KV cache: not filled in yet
  Runtime / allocator overhead: not filled in yet
  Operating system / services: not filled in yet
  Other processes: not filled in yet
  Unallocated reserve: not filled in yet

For another model, change PARAMETERS to its parameter count. Replace each None with the decimal GB you measure on your device, and the script adds the rows up.

Other checks for local inference

Once the budget fits, we confirm that the API path works for the agent and trace where every process sends its traffic.

The local API response shows how that path handled the request. One response does not cover every tool-call or streaming behavior a coding agent needs, so we also run a harmless tool call over the API path we chose and check that the harness finishes it.

The inference request may stay on the local network while model downloads, authentication, updates, or a tool action reach other destinations, so we check each process's traffic destinations separately from the model server's address. The model server and the model files it downloads are attack surfaces of their own, which Model servers are web apps and Model files are code cover. The table lists which component owns each path.

Traffic paths by process.

Sources: Serve LLMs with vLLM on One Device | DGX Spark (vllm) Serve LLMs with vLLM on One Device | DGX Spark (instructions).

Traffic / execution classHow we trace it
Model and container downloadsWe inspect the selected downloader's configuration and observe its process and destinations.
AuthenticationWe identify the client or service, its effective auth mechanism and its observed destination, without reading secret contents.
TelemetryWe read the current configuration and observe network traffic per process.
UpdatesWe read the updater's configuration and observe its owning process and destination.
Agent API requestWe confirm the coding agent's exact process on the host, then check its effective endpoint and a request trace. Serve LLMs with vLLM on One Device | DGX Spark (vllm) Serve LLMs with vLLM on One Device | DGX Spark (instructions)
Agent toolsWe note the allowed actions, identify the tool executor and observe where it sends traffic; destinations depend on the tool.

Two long-context requests and the 4-bit estimate

We finish by applying the steps to one concrete target.

The 4-bit row gives 16.4 GB of raw parameter payload, and suppose we want Qwen3-32B to serve two long-context requests on the 128 GB Spark. To check whether it fits, we use a supported checkpoint and measure allocations at the target context and concurrency, including cache, execution overhead, and system demand.

A DGX Spark, 128 GB of unified memory, beside a tall memory bar named Installed memory. The bottom of the bar is filled and labelled Weights at 4 bits: 16.4 GB. Beside the bar two empty outlined blocks are tagged Long request 1: unmeasured and Long request 2: unmeasured, and a dashed outline names Two long requests. The filled part is to scale; the outlines are not.
  1. Installed memory: one DGX Spark with 128 GB of unified memory.
  2. Raw parameter payload: 32.8 billion parameters at 16 bits take 65.6 GB.
  3. One request: its KV cache and working memory sit on top of the weights, and only a runtime measurement gives their size.
  4. A second long request: the payload estimate stays the same while the runtime holds two long requests.
  5. The 4-bit estimate: the payload drops to 16.4 GB, and the rest of the budget is still measured on the device.
The 4-bit payload is arithmetic from the model card; what the two long requests add is measured on the device.

We also need to check coding quality on the same serving config to see how well it supports a coding agent.

The one thing to take away

The raw payload is arithmetic from the model card and installed memory comes from NVIDIA's specification; the remaining lines of the budget come from measuring the chosen serving configuration on the device itself, plus the reserve you declare.

What an attacker targets here

One Spark serving models runs two things an attacker wants: an HTTP server that manages models, and checkpoint files that can run code when loaded.