Designing a Platform for AI Assistants

This is a thought exercise I keep coming back to, because it pulls together almost everything I’ve been reading about platforms.

The scenario

Imagine an organisation where several teams build AI assistants. Every new assistant needs a repository, a deployment pipeline, an evaluation harness and monitoring, and every team builds those from scratch.

A first cut using domain-driven design (Where Platforms Fit: DDD Subdomains):

• Supporting: the scaffolding (repo, pipeline, eval harness, monitoring). Needed, fairly specific, not differentiating. A platform should offer it as a paved road.
• Core: the conversation design and domain knowledge inside each assistant. That’s what makes each one valuable, so it stays with the teams.
• Generic: model hosting. Buy it.

That framing is mostly right. Here’s how I’d sharpen it, then how I’d actually get it built.

Sharpening the boundaries

1. Evaluation is half supporting, half core. The harness (runners, scoring, regression gates, dashboards) is supporting. The eval datasets and the definition of a good answer are domain knowledge, and they belong to each team. The platform should make plugging in a team’s evals trivial, not own them.
2. Buy the models, but keep a seam. Model hosting is a commodity, yet assistants shouldn’t call providers directly. A thin gateway that maps “the kind of model I need” to a concrete provider is a classic lookup translation (Cloud Platform Translators). It buys switching, routing, rate limits, cost control and a single audit trail.
3. Add the missing layer: guardrails and compliance. Assistants in regulated domains need defaults for personal data, safety filtering, data residency and audit. This is the relevance gap in action: generic scaffolding isn’t enough. Domain-aware defaults are often the most valuable thing a platform can offer, and the thing teams are worst at rebuilding.
4. Observability means more than monitoring. LLM systems need traces of prompt, tool calls and response, cost per conversation, latency, and drift in eval scores over time (Observability).
5. Prompts are code. Version, review and roll back prompts and model configuration like any other change. A prompt edit can break an assistant as surely as a deploy can (Blue-Green Deployment).

Winning buy-in

Different people need different evidence:

• Leadership needs the cost of duplication: how many teams rebuilt each piece, roughly how long it took, and the risk of a different guardrail setup in every team. Frame it as a business case: time to first assistant, compliance exposure, one bill for model spend.
• Teams need relief, not a mandate. Golden paths win by being the easiest option. Be explicit that their prompts, domain logic and evals stay theirs.
• Security and compliance usually become the strongest allies, because one audited path beats a dozen.
• A lighthouse team does more than any slide. Partner with a team about to start a new assistant, build the path with them, and let their result make the case.
• A public roadmap buys goodwill from the teams whose request isn’t first.

Discovering what to build (explore)

This is the phase teams rush and shouldn’t (Explore, Expand, Extract):

• Shadow three to five teams. Map the real path from idea to production. Where were days lost? What got copied between teams?
• Inventory before inventing. The best internal solution often already exists in one team. Elevate it, and don’t flatten teams that already have mature setups.
• Rank friction by how often it happens, how much it hurts and how much risk it carries. I’d expect pipelines and environments, eval regression gates, guardrails, observability and cost, and repo templates near the top.
• Find the base cases. Most assistants probably fit two or three shapes, such as retrieval-based Q&A, tool-using agents and internal copilots. Template those, not every edge case.
• Beware “I’d use it if you just built X”. Chasing edge requests spreads a platform thin (Growing a Platform: Breadth, Depth and Gear Shifting).

Designing the plan

• The thinnest viable platform first: a template repository, a pipeline with eval gates, the model gateway, and tracing and dashboards by default. Guardrails follow as soon as the first regulated team arrives.

• Map it to the planes of an Internal Developer Platform:

  • management plane: a CLI or portal where “new assistant” creates the repo, pipeline and environments
  • service plane: runtime, eval runner and gateway
  • control plane: infrastructure as code that keeps environments reconciled

• Optional but attractive, with a gear shift. Teams can drop down to a provider’s SDK when they need something the path doesn’t offer. They just own the extra work.

• Measure what it enables:

  • time from idea to first assistant in production
  • share of assistants on the paved road
  • eval coverage on every release
  • incidents and compliance findings
  • developer satisfaction and DORA Metrics

  Use these to learn, never as targets (Goodhart’s Law).

Building it (expand)

• Build with the lighthouse team, on their deadline. Their success is the demo.
• Templates backed by packages. Keep shared logic in versioned packages, so improvements reach every assistant and templates don’t drift into forks. It’s the main lesson from building a Web Accelerator.
• Make the eval gate deterministic. No release goes out if it regresses on the team’s own eval set. It’s the automated gauntlet from Don’t Read the Code? Verifying AI-Written Code, applied to assistants.
• Self-service from day one. If creating an assistant needs a ticket, you’ve built a shared-services team, not a platform.
• Docs as you go: defaults, a “your first assistant in a day” guide, and short tips that point to advanced features.

Iterating (and only later, extracting)

• Spiral upwards: each cycle, add a little to each layer, re-interview users, and publish what changed.
• Treat routing-around as a finding. If teams bypass the path, that’s discovery data, not a compliance problem.
• Design for late joiners. The tenth team meets the whole platform at once, so the base case must stay simple.
• Extract when it’s proven: stable adoption, clear savings, and a clear understanding of why it works. Then standardise, publish the business case and scale it with more assistant shapes and compliance packs. Don’t let pressure to prove a return push you into extracting before teams genuinely depend on it.

Failure modes to watch

• The cathedral: a year and a half of building, and no users at launch.
• Over-abstracting the models: if teams can’t reach model-specific features, they’ll bypass the platform.
• Owning teams’ prompts or evals: that’s their core.
• Metrics turning into targets.

The training version

It’s a skill tree: unlock the fundamentals (template, pipeline, gateway) before the advanced moves (guardrail packs, multi-agent shapes), and fix form before adding load. A platform that skips the basics plateaus the same way a lifter does.

Related: Platform Strategy · Domain-Driven Platform Engineering · Thinking in Platforms · What Makes an AI Agent