A reference architecture for an enterprise AI platform
My opinionated answer to "how would you build this for an enterprise" — a gateway, guardrails, governance, evals, and cost control treated as load-bearing parts of the system, not things bolted on after the first incident. This is a design artifact, not a description of a system running in production. Select a stage to see the decision behind it and the alternatives I'd reject.
Request path
Applies to every request
Edge gateway
Single entry point for every model call — auth, rate limiting, and routing live here, not scattered across services.
Why it exists
Every consumer (an internal app, an agent, a batch job) talks to one stable interface regardless of which model or provider sits behind it. Swapping a provider becomes a routing-table change, not an application redeploy.
Rejected alternatives
Let each service call model providers directly
Auth, rate limits, and cost controls would need to be reimplemented per service, and a provider outage or key rotation becomes N incidents instead of one.
Ship a shared SDK library instead of a network gateway
A library still needs a new version shipped to every consumer on every policy change; a gateway changes behavior for all consumers by deploying once.