Decide what we need the extra layer to do

Every layer we put between a coding agent and a model provider is another service to run and another place that can retry or stop waiting.

Our coding client can already reach a provider directly, and the team wants every request to go through a gateway. We identify whether we need translation or shared controls before taking on more configuration and another service to run. By the end we can name the job the extra layer must do, choose a direct connection, an adapter, or a governance gateway for that job, and check timing, retries, and licensing for each service we add.

A direct connection may already do it. An adapter fixes one API mismatch, while a gateway provides shared identity, routing, budgets, or logs and may handle the translation as well.

We work through seven steps in order:

  1. Name the jobs we need from the layer.
  2. Try a direct connection first.
  3. Add an adapter for one specific mismatch.
  4. Use a gateway when shared controls need one place.
  5. Give each layer on the path a separate job.
  6. Count attempts within the caller's wait time.
  7. Check the edition and license for each feature.

Separate the jobs before choosing a product

A proxy or gateway can do up to four jobs, and we assess each product against those jobs.

An API adapter can accept a request and still drop a tool call or a stream event. An access-control gateway is also a service someone has to run, with its own credentials and data handling. The four jobs are these:

  1. Protocol compatibility covers the request, response, stream, and error contract.
  2. Authentication decides which credential represents the caller.
  3. Routing picks the upstream destination.
  4. Governance applies shared controls such as model access, spending limits, or usage attribution.

One component can do several of these jobs, and we need more than its name or port number to know which ones are enabled.

Try a direct connection when it supports what we need

Say three things hold:

We start with the direct path.

The client builds the request, its native adapter sends it to the provider, and the reply comes back through that adapter to the client's orchestration code (the harness). So a text case and a tool case tell us whether the behavior matches.

Protocol compatibility and authentication get separate checks. Codex's custom-provider authentication reference lists these authentication options:

A coding agent and a panel named Client configuration, with three rows (endpoint, model ID, authentication), stand on the client's side; a model provider stands inside its own boundary. Between them is an empty dashed outline labelled No extra service here. Two lanes connect agent and provider: 1 Request, 2 Reply. At the end one pale card rests at the agent.
  1. Client configuration: we start with the direct path when the client needs one provider, streamed text and a harmless tool round trip, its native adapter supports the provider's contract and credential method, and nobody needs a shared policy.
  2. Request: the client builds the request and its native adapter sends it to the provider.
  3. Response: the reply comes back through that adapter to the harness.
Direct access through the client's own configuration.
Figure text

Two actors, an agent with its native adapter and a provider API, exchange a request and response. The client configures the endpoint and authentication and owns local configuration; the provider owns its API. The model has no filesystem access here.

  • The direct path, no extra layer
  • Dashed outline: no extra service
  • This is the path to try first. A text request and a tool request then show whether it behaves as needed. This figure shows the path, not a test result.
  • 1 Request
  • 2 Reply
  • Coding agent: Its native adapter sends the request
  • Client configuration
  • No extra service here
  • Model provider: Provider API
  • Client
  • Provider boundary
  • Endpoint; Model ID; Authentication
  • The reply comes straight back to the agent.

Sources: [1] Authentication | ChatGPT Learn.

Add an adapter for a specific mismatch

An adapter earns its place when the only change is one API mismatch between the client and the provider.

Suppose the client sends OpenAI-compatible requests to a provider endpoint expecting a different contract. An adapter must then support both the client-facing API and the provider's upstream API, and we check that it supports the actual API surface and the features we need. The LiteLLM AI Gateway is one example. Applications send OpenAI-compatible requests to its self-hosted server, which forwards them to the chosen provider and returns an OpenAI-format response.

Both directions need a test. The adapter converts the client's request on the way up, and the reply passes back through it before the client can use it. We record the gateway's upstream credential and retry policy in the route description, including when we added the gateway only for format conversion.

Three stations: a coding agent, LiteLLM as a format adapter inside a proxy process, and a model provider inside its own boundary. Four lanes in order: request in the client's format, request in the provider's format, reply in the provider's format, reply in the client's format. At the end one pale upright card, tagged Reply, rests at the agent.
  1. Client request: the application sends an OpenAI-compatible request to the gateway.
  2. Provider request: suppose the provider endpoint expects a different contract: the adapter converts the request on the way up and forwards it to the chosen provider.
  3. Provider response: the reply passes back through the gateway.
  4. OpenAI-format response: the gateway returns an OpenAI-format response to the client.
The gateway passes the provider's response back to the client.
Figure text

An OpenAI-compatible client sends a request through LiteLLM to a provider, which returns its response through LiteLLM to the client. The proxy and provider API form separate boundaries.

  • An adapter for one mismatch
  • Client's format (upright, round)
  • Provider's format (sideways, cut)
  • Two conversions, two tests. The adapter also holds its own provider credential and retry settings, so those go in the route description.
  • 1 Request, client's format
  • 2 Request, provider's format
  • 3 Reply, provider's format
  • 4 Reply, client's format
  • Coding agent: Client, sends OpenAI-compatible requests
  • LiteLLM: Format adapter
  • Model provider: Expects a different format
  • Reply
  • Proxy process
  • Provider boundary
  • The adapter converts the reply so the client can use it.

Sources: [1] LiteLLM AI Gateway | liteLLM.

One request traced through LiteLLM

Following one Anthropic Messages request through LiteLLM to a GPT model and back shows what the adapter changes in each direction.

Three stations: a coding agent, LiteLLM as a format adapter, and a model provider. Four lanes connect them in order: 1 Request, Messages format. 2 Request, Responses format. 3 Tool request, Responses format. 4 Tool request, Messages format. At the end one pale card tagged Reply: tool request rests at the coding agent. No tool has run.
  1. Agent to LiteLLM: an Anthropic Messages request to /v1/messages.
  2. LiteLLM to provider: a Responses request to /openai/v1/responses.
  3. Provider to LiteLLM: HTTP 200 with one function_call item.
  4. LiteLLM to agent: a Messages reply with a tool_use block.
The request below changes API on the way up and shape on the way back.

1. What the client sent

{
  "model": "gpt-route",
  "max_tokens": 256,
  "tools": [
    {
      "name": "get_weather",
      "description": "Get the current weather for a city.",
      "input_schema": {
        "type": "object",
        "properties": {
          "city": {"type": "string", "description": "City name"}
        },
        "required": ["city"]
      }
    }
  ],
  "messages": [
    {"role": "user", "content": "What is the weather in Paris right now?"}
  ]
}

The client posted this body to the gateway's /v1/messages with x-api-key: $LITELLM_KEY and anthropic-version: 2023-06-01. It is a plain Messages request: the model is the route alias gpt-route, and the tool's schema sits in input_schema.

2. What LiteLLM sent upstream

{
  "model": "gpt-6-luna",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [
        {"type": "input_text", "text": "What is the weather in Paris right now?"}
      ]
    }
  ],
  "include": ["reasoning.encrypted_content"],
  "max_output_tokens": 256,
  "tools": [
    {
      "type": "function",
      "name": "get_weather",
      "strict": false,
      "description": "Get the current weather for a city.",
      "parameters": {
        "type": "object",
        "properties": {
          "city": {"type": "string", "description": "City name"}
        },
        "required": ["city"]
      }
    }
  ]
}

LiteLLM posted this body to /openai/v1/responses, the upstream Responses API, not to Chat Completions. Along the way it made these changes:

3. What the upstream returned

{
  "id": "resp_...",
  "object": "response",
  "status": "completed",
  "model": "gpt-6-luna",
  "output": [
    {
      "id": "fc_...",
      "type": "function_call",
      "status": "completed",
      "arguments": "{\"city\":\"Paris\"}",
      "call_id": "call_...",
      "name": "get_weather"
    }
  ],
  "store": true,
  "usage": {"input_tokens": 57, "output_tokens": 18, "total_tokens": 75}
}

The Azure OpenAI deployment answered HTTP 200 with one function_call item, its call_id, and the arguments as a JSON string. Usage was 57 input and 18 output tokens, and the reply reported "store": true although the request did not set store.

4. What LiteLLM returned to the client

{
  "id": "resp_...",
  "type": "message",
  "role": "assistant",
  "model": "gpt-route",
  "stop_sequence": null,
  "usage": {"input_tokens": 57, "output_tokens": 18},
  "content": [
    {
      "type": "tool_use",
      "id": "call_...",
      "name": "get_weather",
      "input": {"city": "Paris"}
    }
  ],
  "stop_reason": "tool_use",
  "stop_details": null
}

LiteLLM rebuilt the call as a Messages tool_use block with the same call_... id and the arguments parsed into an input object, set stop_reason to tool_use, and carried the usage over. The model reads gpt-route because LiteLLM replaced the upstream model name with the requested alias, as its log line below states. The id is resp_ followed by base64 that decodes to litellm:custom_llm_provider:openai;model_id:...;response_id:resp_..., so the client can read the provider and the upstream response id.

The log lines at each boundary

The --detailed_debug log marks each boundary; we kept the lines that mark one, removed the timestamps, the recorder's local address and the call id, and cut the long lines:

LiteLLM:DEBUG: utils.py:550 - litellm.anthropic_messages(..., api_base='.../openai/v1', ...)
LiteLLM:DEBUG: utils.py:550 - litellm.aresponses(model='openai/gpt-6-luna', input=[{'type': 'message', 'role': 'user', 'content': [{'type': 'input_text', 'text': 'What is the weather in Paris right now?'}]}], include=['reasoning....
POST Request Sent from LiteLLM:
.../openai/v1/responses \
-H 'Content-Type: application/json' -H 'Authorization: Be****ey' \
LiteLLM Router:INFO: router.py:5550 - ageneric_api_call_with_fallbacks(model=openai/gpt-6-luna) 200 OK
LiteLLM Proxy:DEBUG: common_request_processing.py:1449 - litellm_call_id=...: response model mismatch - requested='gpt-route' downstream='gpt-6-luna'. Overriding response['model'] to requested mod...
INFO:     "POST /v1/messages HTTP/1.1" 200 OK

The log printed the prompt text in full, while the upstream Authorization header appeared masked as Be****ey.

We captured this trace with LiteLLM 1.106.4 from PyPI in a python:3.12-slim container started with --detailed_debug. Its config had one route, gpt-route, pointing at openai/gpt-6-luna through a small recorder on the host, which forwarded each call to an Azure OpenAI deployment. The recorder added the upstream key to each call, so the key never entered the container. The blocks show request and response bodies only, the ids are shortened, and the upstream reply keeps only its id, object, status, model, output, store and usage fields.

Source: /v1/messages | liteLLM. LiteLLM documents that /v1/messages takes and returns the Anthropic Messages format for all its providers, including openai; the choice of the upstream Responses API and the field changes above are what we saw in our run.

The trace shows why an adapter needs a test in each direction: the request changed API and field names on the way up, and the reply was rebuilt into another shape on the way back. It also shows that the gateway reads every request and reply it converts, and with debug logging on it writes the prompt to its own log. What an attacker can do from that position, from stealing provider keys to reading stored prompts, is the subject of The gateway as an attack surface.

Shared controls can justify a gateway

Once the team on the direct path needs to attribute usage and enforce its budget centrally, with the protocol unchanged, a gateway can enforce those shared controls in one place.

LiteLLM's gateway feature list includes virtual keys scoped by user, team, or project, with model access, budgets, and rate limits. It also lists:

We need to enable those features in our deployment.

Kong's AI Gateway overview describes centralized provider credentials, model access and token budgets by group, and prompt and response policies. We check which product generation, version, and edition provides each feature.

The gateway needs an owner. That person handles:

Because the gateway sees every request and reply, we have to decide what it logs and how to configure it before any traffic flows through it.

For Claude Code, third-party gateways operate without Anthropic's endorsement, maintenance, or audit. Routing Claude Code to non-Claude models through any gateway is unsupported by Anthropic.

Give each of the two layers a separate job

When our governance gateway already translates the client's contract into the provider's, we need a separate reason to add another adapter.

The starting path is client to gateway to provider, with replies coming back the same way. We add an adapter between the gateway and the provider only if a mapping or feature we need is still missing, and then we test every interface on that path.

A coding agent on a platform labelled Client owner, a governance gateway with a panel of configured controls (identity, budgets, logs), an optional format adapter, and a model provider. After the gateway a branching point asks Does the gateway already convert? and has two exits: A, the gateway already converts, a solid lane to the provider; B, a mapping or feature is missing, dashed lanes through the optional adapter. At the end a card tagged Request, option B rests at the provider.
A governance gateway with an optional adapter.
Figure text

A client uses a governance gateway, with an additional adapter only for a remaining required mapping or feature the gateway cannot supply. The gateway operator, adapter operator, and provider have separate ownership boundaries. Dashed lines mark optional paths.

  • A second layer needs a reason
  • Optional part (dashed, outline)
  • Each layer has its own job and its own operator. Replies come back the same way. Dashed parts are options, not parts we must run.
  • 1 Request
  • 2 Request, past the shared controls
  • A, the gateway already converts
  • B1: B, a mapping or feature is missing
  • B2: Converted request, if supported
  • Coding agent
  • Governance gateway
  • Configured controls
  • Does the gateway already convert?
  • Optional format adapter
  • Model provider
  • Client owner
  • Identity; Budgets; Logs
  • Request, option B
  • The longer path has more hops, and each hop gets a test.

Sources: [1] Other LLM gateways - Claude Code Docs; [2] LiteLLM AI Gateway | liteLLM; [3] Kong AI Gateway | Kong Docs; [4] AI Proxy Advanced - Plugin | Kong Docs.

Count attempts within the allowed wait time

The layers we choose affect request duration, so we count time and attempts along the path.

We check where each added service can retry or stop waiting. Track provider attempts separately from the waits between them.

We compare total elapsed time with the caller's timeout, then check separately whether cancellation reaches the provider. The provider call keeps running if the client stops waiting without cancelling it. Retries at several layers can also produce more provider attempts than the client sees as requests, and the provider and gateway terms decide whether those extra attempts are billed or deduplicated.

A coding agent, a gateway and a model provider, with a branching point after the provider, After an attempt, that has three exits. A retry lane loops back to the gateway. A time bar named Time the caller waits is filled past a marked limit, and its top part is red: past the caller's limit. A counter beside the provider reads Attempts 2, client requests 1. At the end the request card rests at the model provider. No amounts are shown; the budget is symbolic.
The budget includes handling, provider attempts and retry waiting.
Figure text

The symbolic request path shows gateway handling, a provider attempt, a direct success exit, and an optional retry loop to another attempt before an exhausted exit. The budgets are symbolic. Whoever configures a retry owns its policy.

  • Count attempts inside the wait
  • Past the caller's wait limit (bar)
  • Add up handling, attempts and waits, compare the total with the caller's wait limit, and check separately whether a cancel reaches the provider. The amounts here are symbolic.
  • 1 Request
  • 2 Attempt
  • 3 Outcome of the attempt
  • R: Retry, wait, then try again
  • Time the caller waits: Provider attempt
  • Attempts 2, client requests 1
  • Past the caller's limit
  • Coding agent
  • Gateway
  • Model provider
  • Provider attempts
  • After an attempt
  • The caller's wait runs out. If nothing cancels the provider attempt, it keeps running.

Sources: [1] LiteLLM AI Gateway | liteLLM.

T_elapsed = C + Σ(i=1..N)(G_i + P_i) + Σ(i=1..N−1)B_i + R.

For N ≥ 1 serial provider attempts: C is client handling, G_i is gateway handling for attempt i, P_i is provider attempt time, B_i is the wait/backoff before the next permitted attempt, and R is response transit. The backoff sum is zero for N = 1; there is no backoff after the final attempt. Each elapsed interval is counted once. Nested or parallel retries are outside this symbolic sequence. Each term uses the same duration unit, such as seconds.

Compare a completed path's elapsed time with the caller's wait policy. An exhausted exit can return an error or timeout.

Symbolic elapsed-time stages
Stage Elapsed-time term Retry policy Exit
Client handling C Caller owns its wait policy Request begins, or caller stops waiting
Gateway handling Σ G_i Gateway owns any configured retry policy Forward a permitted attempt
Provider attempts Σ P_i Only eligible failures may be retried Success, eligible retry, or exhausted exit
Retry waiting / backoff Σ B_i (i = 1..N−1) Only before another permitted attempt Next attempt or exhausted wait budget
Response transit R No further retry assumed in this sequence Caller consumes the response if still waiting; late responses may arrive and be discarded.

LiteLLM gateway retry settings (LiteLLM AI Gateway | liteLLM).

Check the edition and license separately

Gateway feature lists leave out which edition or release ships each feature, so we read the license and plugin pages as well.

The LiteLLM license applies MIT terms outside the enterprise directory, subject to its stated restrictions. Content under enterprise/, if present, has its own license.

Kong's AI Proxy Advanced plugin is part of AI Gateway Enterprise. The plugin page describes translating standardized OpenAI-format requests to a configured target and transforming replies back, with load balancing, retries, and fallback. Some behavior is version-specific, including broader cross-format fallback from v3.10 onward, and the page also links a separate policy for AI Gateway 2.0.

Selected product and path details.

Sources: Authentication | ChatGPT Learn, Other LLM gateways - Claude Code Docs, LiteLLM AI Gateway | liteLLM, litellm license, Kong AI Gateway | Kong Docs, AI Proxy Advanced - Plugin | Kong Docs.

Product or pathProtocolIdentity and budgetsRouting and observabilityLicense or edition
Direct native connectionThe protocol must match the client/provider contract.Identity and budgets are provider-specific.The connection provides no shared gateway behavior.The client's and the provider's own terms apply.
LiteLLM AI GatewayThe gateway uses an OpenAI-compatible request and response path; exact features depend on the release.Virtual keys, budgets, and rate limitsBalancing, fallback, retries, spend tracking, and logging integrationsMIT terms apply outside the stated enterprise scope; the enterprise directory has its own license.
Kong AI Gateway overviewProtocol translation comes from the AI Proxy Advanced plugin in the next row.Central credentials, group access, and token budgetsGovernance capabilitiesEntitlements vary by edition and version; AI Proxy Advanced is part of AI Gateway Enterprise
Kong AI Proxy AdvancedThe plugin translates configured formats; native-format and fallback behavior vary by version.The policy we configure sets identity and budget controls.Load balancing, retry, and fallbackAI Gateway Enterprise only; some behavior is version-specific.

Choose the layer for its job

Every layer between a coding agent and a provider needs a named job and a test, and we check its edition and license. Our next route decision starts with the first of the seven steps above.

What an attacker targets here

The LiteLLM route we traced puts one process between the agent and the provider, and an attacker looks at three things in it.