How I Put 100 OpenClaws Behind One LLM Gateway

When I wrote about hosting more than 100 OpenClaw instances, the isolation boundary was the point: one container, volume, network, configuration, and credential set per customer. That is still the right boundary for customer data and tools. It is a poor boundary for model subscriptions.

The fleet has several ways to reach a capable model: two Opus subscriptions, a Kimi coding plan, and MiniMax as the final backstop. Copying those credentials into every container made each claw responsible for concerns it could not handle well. Changing priority meant rewriting fleet configuration. A rate-limited subscription failed independently in every container. A compromised claw could reach a provider directly. And there was no authoritative answer to a basic operational question: which claw used the tokens?

The gateway fixes that by separating two jobs. OpenClaw decides what to ask a model. The gateway decides which subscription should answer, whether that subscription is healthy, and whether this claw still has budget to use it.

One stable API, four different upstreams

The public shape of the gateway is deliberately boring: POST /v1/messages, using the Anthropic Messages protocol OpenClaw already speaks. A claw does not know whether Opus A, Opus B, Kimi, or MiniMax ultimately answered. It sends the same request with its tenant key and receives the same kind of response.

That does not mean the gateway normalizes everything into a lowest common denominator. The two Anthropic subscriptions and MiniMax’s Anthropic-compatible endpoint are passed through directly. Headers, usage blocks, tool calls, and server-sent events retain their native shape. Only Kimi’s coding endpoint requires translation, because that plan exposes an OpenAI-style chat API. The gateway translates the request, tool definitions, response, and streaming events at that one edge.

Inside each OpenClaw, the provider configuration became small enough to reason about at a glance:

models: {
  providers: {
    gateway: {
      baseUrl: "http://model-gateway:8790",
      apiKey: "${MODEL_GATEWAY_KEY}",
      api: "anthropic-messages"
    }
  }
}

agents: {
  defaults: {
    model: {
      primary: "gateway/claude-opus-4-8",
      fallbacks: ["gateway/claude-sonnet-5"]
    }
  }
}

The model names stay useful to OpenClaw, but the actual chain is a gateway concern. Most importantly, a gateway tenant has no direct Anthropic, Kimi, or MiniMax credential. The only secret in the claw is a revocable key scoped to that claw.

Routing for cache locality first, failover second

The first two upstreams form a sticky group. A consistent hash of the tenant chooses which Opus subscription that claw starts with. Tenants distribute across both subscriptions, but one tenant normally returns to the same one. That keeps Anthropic’s prompt cache warm instead of randomly spraying a long-running worker’s context across two accounts.

Stickiness is a preference, not a promise. Before the first response byte, a transport error, authentication failure, timeout, rate limit, or upstream server error advances the request through the chain. If an upstream fails repeatedly, its circuit breaker opens with a doubling cooldown. A provider 429 uses its own Retry-After value. One claw discovers the problem; the rest skip the unhealthy subscription.

There is also an in-flight ceiling for every upstream. Once a subscription is full, new work tries the next one instead of joining an unbounded queue. If the entire chain is saturated, the gateway returns a useful 429 with Retry-After. Backpressure is much easier to operate when it is explicit.

After streaming begins, the gateway never swaps models halfway through an answer. That would risk duplicating tool calls and splicing two different responses together. An in-stream failure is returned in protocol and OpenClaw’s next retry starts against the next healthy upstream.

A quota is part of routing, not a separate throttle

The important quota decision was to budget each claw × upstream pair, not just the claw’s total traffic. Input and output tokens count together. The default window is five hours, beginning with that tenant’s first counted request and resetting when the window elapses. The counters survive gateway restarts, so restarting the container does not accidentally refill everybody’s allowance.

When a claw reaches its limit on Opus A, it has not reached a dead end. Opus A simply becomes unavailable to that claw for the rest of its window, exactly as if a per-tenant circuit breaker had opened. The gateway tries Opus B, then Kimi, then the uncapped MiniMax backstop. Other claws can continue using Opus A because their budgets are separate.

This is why quota enforcement belongs inside the router. A global middleware can say no. The gateway can say, “not this subscription for this tenant right now,” and still complete the request somewhere appropriate.

The defaults are intentionally ordinary rather than clever: one million combined tokens per five-hour window on each of Opus A, Opus B, and Kimi, with MiniMax available as the uncapped final path. An override can replace one number, leave another at its fleet default, or make a particular subscription unlimited for one claw.

The fleet page is the control plane

Quota machinery is not useful if changing a limit requires shell access to the gateway host. I added the controls to the same account Fleet page I already use to watch worker health and telemetry.

The grid answers the first question quickly: is the fleet healthy? Green, warning, error, and never-reported states are visible without opening 100 cards. Selecting a worker connects that fleet-level signal to its model consumption.

The same panel shows 24-hour, seven-day, and 30-day token totals, split into input and output, followed by model-level usage and the latest heartbeat. The quota editor uses three values deliberately: blank means “inherit the fleet default,” a number replaces the default, and unlimited removes the cap for that upstream. Clearing the form returns the claw to fleet policy.

Saving an override appends a claw_quotas_set event to the website’s existing event stream. The gateway polls an authenticated quota feed every five minutes and atomically replaces its in-memory table. If the website is temporarily unreachable, the gateway keeps the last valid table rather than turning quotas off. The account site remains the source of truth; the model path does not depend on it being reachable for every request.

The credential boundary matters more than the proxy

A reverse proxy is easy. The security property I wanted was harder: once a claw moves to the gateway, it must not retain a route around the gateway.

Each tenant key is generated with the tenant name and random material, printed once, and stored server-side only as a SHA-256 hash. The key identifies the claw for routing counters, quotas, and the tenant status endpoint. It cannot be used as an admin key and it cannot authenticate directly to any model provider.

During cutover, the provisioner removes direct model credentials from the container environment and puts them in host-side escrow. OpenClaw’s provider list becomes gateway-only, which also prevents an old stored provider profile from silently taking precedence over the new key. When an unassigned worker is suppressed, its gateway key is escrowed too and restored on assignment.

The distinction is useful operationally. Rotating an upstream subscription now happens once at the gateway. Revoking one worker means deleting one tenant key. Neither operation requires distributing a provider secret across the fleet.

I made the migration boring on purpose

A fleet-wide model change is exactly where a half-written configuration can strand workers. The control plane therefore treats “this tenant should use the gateway” and “this tenant is ready to use the gateway” as different states.

1. Set the tenant’s gateway URL. Its container joins the shared openclaw-model-gateway Docker network, but direct model wiring remains live.
2. Mint that tenant’s gateway key and place it in the tenant environment.
3. Apply the tenant. Only when the URL and a real key are both present does the renderer switch to the gateway-only provider configuration and escrow the direct credentials.
4. Verify a real streamed answer, the x-mgw-upstream response header, the tenant status endpoint, and the new telemetry on the Fleet page.

Rollback is the inverse control-plane command: turn the gateway flag off and apply the tenant. The provisioner restores the escrowed direct credentials and renders the previous provider chain. The design does not rely on somebody editing generated JSON inside a running container.

What became easier

Provider failure is now a fleet event. Circuit breakers and Retry-After handling are shared. Once one request discovers a bad upstream, every claw benefits.

Usage has an owner. Request, error, input-token, and output-token counters are stored by tenant and upstream. Every response says which upstream served it, and each claw can read only its own status.

Fair use is enforceable without making the fleet brittle. A heavy claw exhausts its own budget on one subscription and moves down its own chain. It does not empty another claw’s allowance or force a blanket outage.

The provider boundary is finally explicit. Claws hold tenant credentials; the gateway holds subscription credentials. OpenClaw remains isolated per customer while model capacity is pooled and governed centrally.

Adding another upstream is a gateway change. If its protocol matches, it is mostly configuration. If it does not, translation lives at one edge instead of inside every tenant.

The deeper lesson is the same one I learned while consolidating the compute fleet: share infrastructure where sharing improves operations, and keep the customer boundary where isolation matters. For compute, the customer boundary is a container. For model access, it is a tenant key, a set of counters, and a quota table at one deliberately narrow gateway.

Frequently asked questions

Why put an OpenClaw fleet behind an LLM gateway?

The gateway gives every worker one stable model endpoint and one revocable tenant key. Provider credentials, routing order, failover, accounting, and quotas stay in one service instead of being copied into every worker.

How do per-claw model quotas work?

Each claw has a separate input-plus-output token budget for each upstream over a five-hour window. Fleet defaults apply automatically, an operator can override any claw, and an exhausted upstream is skipped while the gateway tries the next available subscription.

Does the gateway translate every model request?

No. OpenClaw speaks the Anthropic Messages API to the gateway. Anthropic-compatible upstreams pass through directly; only the Kimi coding endpoint needs translation to and from OpenAI chat and streaming formats.

What happens when a model subscription is unavailable?

Before a response begins, authentication errors, rate limits, timeouts, transport failures, and server errors advance to the next upstream. Circuit breakers keep the rest of the fleet away from a failing subscription until its cooldown expires.

Related

• How I host 100+ OpenClaw instances: the compute, browser, network, and tenant-isolation layer under this gateway.
• Event sourcing on SQLite in production: the event stream that records quota policy and drives the rest of the 8Examples control plane.
• Connect OpenClaw to Xero: another gateway boundary, this time keeping OAuth tokens and accounting access outside the claw.