AI API & Agent
Spend Firewall.

That depends on a choice you make, and both answers are worth stating. By default — pass-through mode — your provider key travels in the request header, is used once to reach the provider, and is discarded; nothing about it is written to our database. If you opt into the Provider Secret Vault, you are deliberately asking us to hold it, and we store it as AES-256-GCM ciphertext with a 96-bit random nonce, decrypted only in memory to sign an outbound call. In both modes the key is never written to any log, and your LLMCap proxy key is stored only as a bcrypt hash.

Three different credentials, and mixing them up is the most common setup mistake. Your Proxy Key starts tg_live_, is created in the dashboard, and is sent as the X-LLMCap-Key header on every proxied call — it identifies your workspace and is what budgets are enforced against. Your Provider API Key is the Anthropic, OpenAI, Google, Mistral, Cohere or AWS credential you already own: send it per request, or store it in the Provider Secret Vault. A Management API Key starts gk_live_, is optional and advanced, and is used as Authorization: Bearer to call the LLMCap management API from your own scripts — it never carries model traffic. The older header name x-tokenguard-key still works and is deprecated; new integrations should send X-LLMCap-Key.

Yes — streaming is supported from day one. LLMCap passes SSE chunks through in real time and counts output tokens as they arrive. If the budget is exceeded mid-stream, the connection is closed and a final budget_exceeded event is sent. Cutting the stream stops further spend; the tokens the provider already generated before the cut are counted and charged, because the provider produced them.

The next incoming request is refused with HTTP 429 and no model generation is started for that blocked call. One detail worth stating plainly: on Anthropic, LLMCap prices the request through Anthropic's count_tokens estimation endpoint before running the budget check, so that endpoint is contacted even for a request that then gets refused — it returns a count, not a completion. Every other provider is priced locally, with no provider contact at all. What your provider ultimately bills you for is between you and them; what LLMCap controls is whether the generation call is made, and for a blocked request it is not. Your app receives the same 429 response structure providers use for rate limiting, so existing error handling works as-is. Calls that were already in flight when the cap was reached still finish and settle — see the next question.

Yes, by roughly the cost of the responses already in flight — and it is worth being precise about why. LLMCap checks your budget before forwarding each request, so once the cap is reached nothing new is sent to the provider. But the cost of a response is not known until it exists: a non-streaming call is only settled when it returns, so a single unusually long response can carry you a little past the line, and several calls running at once each pass the check before any of them settles. Streaming responses are re-checked as they arrive — every 100 output tokens on Anthropic, OpenAI, Google, Mistral and Cohere, and roughly every 1,000 on Bedrock, where the stream is measured in bytes — and cut at that point, which is what stops a runaway loop from spending thousands. Treat the cap as a hard stop on new spend, not a to-the-cent ceiling. If you need a tighter bound, declare max_tokens on your calls: LLMCap can then account for the worst case up front instead of assuming a default.

Yes, for five of the six — with one condition worth knowing before you rely on it. The Provider Secret Vault stores your Anthropic, OpenAI, Google, Mistral or Cohere key encrypted at rest with AES-256-GCM (96-bit random nonce per entry). Stored keys are never returned after the initial save and never written to logs. The condition: LLMCap reads the stored key only when a request arrives with no provider credential header, because a header that is present always wins. Your client therefore has to be able to issue a request that carries no provider credential header, and the official OpenAI and Anthropic Python SDKs both require a credential in normal use: OpenAI refuses to construct without one, and Anthropic refuses to send the request. The official Google and Cohere SDKs are the same story for their own reasons — Google refuses to construct without an api_key, Cohere requires a token. Passing a fake key does not help either, because we forward whatever you send. The one official SDK that does reach the Vault is Mistral: construct it with no api_key and it sends no Authorization header at all. So the Vault applies to raw HTTP calls, to custom clients, and to the official Mistral SDK — not to a stock OpenAI, Anthropic, Google or Cohere SDK setup. We checked this by capturing the request each SDK builds, at the versions we tested; official SDK APIs change, and no paid provider call was involved. AWS Bedrock is the exception and cannot be stored at all: SigV4 signing needs an access key and a secret access key, and the vault holds one value per provider, so Bedrock callers send the AWS credential headers with each request instead.

Yes — the Policy Engine takes allow and block rules scoped to a provider, a model, a project, or a single API key. Which rules you write decides how the workspace behaves. Write only block rules and everything else keeps working — that is why putting LLMCap in front of a running application changes nothing. Write your first allow rule and the workspace flips to an allowlist: from then on, anything no allow rule names is refused. A block always wins over an allow, however specific the allow is, so a rule can never be narrowed out of existence. Model names match a version suffix automatically (claude-sonnet covers claude-sonnet-4-6) but never a different model (gpt-4o does not cover gpt-4o-mini). The dashboard states which mode you are in above the rule list.

You configure one or more HTTPS webhook endpoints in the dashboard. When an anomaly is detected (spend spike, budget at 80%/95%, cache drop, or error rate spike), LLMCap POSTs a signed JSON payload to your endpoint within seconds. Each payload includes an HMAC-SHA256 signature in the X-LLMCap-Signature header so you can verify it's genuine. Works with PagerDuty, Slack, Teams, or any custom endpoint.

Yes — AWS Bedrock is fully supported including Claude, Llama, Titan, Mistral, and Cohere models via Bedrock. Pass your AWS credentials in request headers alongside your LLMCap proxy key. LLMCap re-signs requests using SigV4 and forwards them to the Bedrock Runtime. Credentials sent this way are used once and discarded — and headers are the only option for Bedrock, because the Provider Secret Vault holds a single value per provider and SigV4 needs a key pair. This is exactly what fills the gap that AWS Budgets leaves open: AWS Budgets has 8-12 hour reporting lag and sometimes doesn't trigger at all — LLMCap refuses the call before Bedrock is invoked, instead of telling you about it hours later.

Self-hosting is on the roadmap. The proxy is open source (FastAPI + Redis). For now, the managed service at proxy.llmcap.io is the recommended path — it is already deployed and running.

Nothing is stored, but it is worth being exact about what happens in transit. To price a request LLMCap has to parse it: it reads the JSON body to find the model, the stream flag, and the message content it needs to count tokens — and on Anthropic it sends that content to Anthropic's own count_tokens endpoint for an exact figure. What it does not do is keep any of it. The request log records only metadata: provider, model, token counts, cost, status, latency, and app version. There is no column in our database that could hold prompt text, system instructions, user messages, or assistant responses — that is a property of the schema, not a policy we promise to follow. You can confirm exactly what we record by opening the Request Log in your dashboard.

Yes. Your application connects to proxy.llmcap.io over HTTPS — there is no unencrypted endpoint to connect to — and LLMCap's outbound connection to each provider (Anthropic, OpenAI, Google, etc.) is HTTPS as well. We deliberately do not advertise a specific TLS version: the version is negotiated between the client and the platform edge, so promising 1.3 on every connection would be a promise we do not control. Provider API keys you send in request headers are never written to disk or any log. Keys you choose to store in the Provider Secret Vault are encrypted at rest with AES-256-GCM, with a 96-bit random nonce per entry.

LLM calls will fail with a connection error until the proxy recovers. We target 99.5% uptime — the figure in our Terms — via Railway with health monitoring and automatic restarts. It is a target we work to, not a measured track record we can show you yet. For applications requiring higher availability, we recommend wrapping LLMCap calls in a circuit breaker with a configurable direct-to-provider fallback that you can activate if needed. A self-hosted high-availability deployment option is on the roadmap.

No. Requests without X-LLMCap-Agent-Id or X-LLMCap-Run-Id headers have no agent or run budget applied. They are tracked against your workspace and API key budgets as normal. You can run mixed traffic (some tagged, some not) in the same workspace.

They are enforced together — the first exhausted limit blocks the request. If your workspace has a $50/day budget, an agent has a $5/day cap, and a run has a $1.00 total cap, any of the three reaching its limit returns a 429 before the provider is called. Set a high workspace limit for overall protection and tighter agent and run limits for individual workflows — all checked atomically before each request.

When the spending cap is hit during a stream, LLMCap sends a terminal SSE event — data: {"error": "budget_exceeded"} — then closes the connection cleanly. Your existing SSE error handler receives this event the same way it receives any provider-side error, so no special handling is required. On billing: the tokens streamed before the cut were really generated by the provider, so they are counted against your budget and appear in your request log. What the cut prevents is the rest of the response — LLMCap re-checks your budget every 100 output tokens (roughly every 1,000 on Bedrock, where the stream is measured in bytes rather than parsed tokens), so a long runaway generation stops there instead of running to its max_tokens.

Multiple layers limit the blast radius. (1) IP Allowlist — even with valid credentials, requests from IPs outside your allowlist are rejected at the gate before any token is consumed. (2) Provider Secret Vault keys are AES-256-GCM encrypted at rest; an attacker with database access cannot extract them without the separate encryption key. (3) Spending caps are enforced at the proxy — once the cap is reached, new requests are refused, so a stolen key cannot run your bill up past it beyond the calls already in flight. (4) The audit log is append-only and tamper-proof at the database trigger level — every action is recorded and cannot be erased. Assign team members the minimum role they need (RBAC), and enable IP allowlisting on production keys to further reduce exposure.