Building an AI Gateway with Apigee (LLM Proxy)

The first LLM feature at most companies ships with a provider API key in an environment variable. A year later there are a dozen teams doing the same thing — no cost visibility, keys sprawled across services, no shared rate limits, and no way to enforce a policy. This is the exact problem API gateways solved for REST a decade ago, and the fix for AI traffic is the same: put a gateway between your apps and the model providers. If you already run Apigee, you don't need a new product — you can build an AI gateway with policies you already know.

(New to the concept? Start with AI gateways: managing LLM traffic in the enterprise for the why; this post is the how, in Apigee.)

What the AI gateway does

A single control point in front of OpenAI / Anthropic / Vertex that centralises the cross-cutting concerns that don't belong in every app:

• Authentication of internal consumers (which team/app is calling?)
• Cost control via quotas and token metering per consumer
• Rate limiting to protect budgets and provider limits
• Logging of prompts/responses for audit and debugging
• Guardrails — payload masking, blocklists, request validation
• Provider routing — swap or fail over between models centrally

Build the proxy

Create an Apigee API proxy whose target endpoint is the LLM provider (e.g. https://api.anthropic.com). Your apps call the Apigee proxy URL instead of the provider directly — so you can attach policies to every request. Keep the real provider key in the target/backend config (or a Key Value Map), never in the client apps.

The policy stack

Attach these policies to the proxy request/response flow:

1. Authenticate the consumer — VerifyAPIKey (or OAuthV2 for OAuth). Each team gets its own app credential, so you can attribute usage.

<VerifyAPIKey name="VA-VerifyKey">
  <APIKey ref="request.header.x-api-key"/>
</VerifyAPIKey>

2. Rate limit — SpikeArrest smooths bursts; Quota enforces a per-consumer budget.

<Quota name="Q-PerApp">
  <Interval>1</Interval>
  <TimeUnit>day</TimeUnit>
  <Allow count="10000"/>
  <Identifier ref="client_id"/>
</Quota>

3. Meter token usage — on the response, extract the provider's token counts with ExtractVariables, then log/emit them so you can bill or alert per team.

<ExtractVariables name="EV-Tokens">
  <Source>response</Source>
  <JSONPayload>
    <Variable name="input_tokens"><JSONPath>$.usage.input_tokens</JSONPath></Variable>
    <Variable name="output_tokens"><JSONPath>$.usage.output_tokens</JSONPath></Variable>
  </JSONPayload>
</ExtractVariables>

4. Log — MessageLogging ships prompts, responses, and token counts to your logging/analytics stack (mask sensitive fields first).

5. Guardrails — use AssignMessage/JavaScript policies to strip or mask PII before the request leaves your perimeter, and to enforce request size or content rules.

Multi-provider routing

Because every app now goes through the gateway, you can route centrally: send different consumers or request types to different providers, or fail over from a primary to a fallback model — all without touching a single client app. Use a RouteRule or a policy that sets the target based on a header or the consumer's product.

Why this is worth it

Once the gateway is in place you get, for free, the things every enterprise AI program eventually needs: a single place to rotate keys, per-team cost dashboards, enforceable budgets, an audit trail, and the ability to change providers without a fleet-wide redeploy. It turns "shadow AI sprawl" into a governed platform.

FAQ

Do I need a special "AI" product in Apigee? No. An AI gateway is a normal API proxy plus the policies above (VerifyAPIKey/OAuth, Quota, SpikeArrest, ExtractVariables, MessageLogging). You're applying familiar gateway patterns to LLM traffic.

How do I meter cost when pricing is per-token? Extract usage.input_tokens / usage.output_tokens from the provider response with ExtractVariables, then log or emit them per consumer and multiply by the model's per-token price.

Does the gateway add latency? A well-configured proxy adds a few milliseconds — negligible next to LLM response times. Disable response buffering if you proxy streaming responses.

Can it handle streaming (SSE) responses? Yes, but turn off buffering on the streaming route so tokens pass through as they arrive rather than being held until the response completes.

Apigee vs a dedicated LLM gateway? If you already run Apigee, reuse it — you get mature policy, security, and analytics for free. A dedicated LLM gateway makes sense mainly if you want LLM-specific features (semantic caching, prompt firewalls) out of the box.

An AI gateway is the difference between a dozen teams each holding a raw provider key and a governed platform with cost control, security, and observability. If Apigee is already your API layer, it's also your AI gateway — no new product required.

More: AI gateways in the enterprise, the Apigee overview, and Apigee policies & security.

Standing up an enterprise AI gateway and want the design reviewed? Let's talk.