AI Platform Engineering

LLM gateway design

An LLM gateway is platform engineering: one paved road so every team gets models fast, and the company keeps control of cost, security and data.

Requirements to pin down first

• Consumers: internal apps, agents, IDE/coding tools, batch jobs. Expected RPS and tokens/day.
• Providers and models: Bedrock (which regions, cross-region inference), direct vendor APIs, self-hosted.
• Data rules: what can leave the account/region, PII handling, retention, logging of prompts allowed or not.
• Who pays: cost centre per team, per application, per user.

Reference architecture

1. Entry: OpenAI-compatible and/or Anthropic-compatible API so existing SDKs work. Deployed on EKS/ECS behind an internal ALB, or a managed option (LiteLLM, Portkey, Kong AI Gateway, Envoy AI Gateway). Build only the parts that are a real differentiator.
2. Identity: one virtual key per application or team, mapped to an owner and cost centre. Prefer workload identity (IRSA/OIDC) over long-lived keys for services.
3. Routing: model aliases (chat-default, code-large) so the platform can swap underlying models; fallback chains across regions/providers on throttling or 5xx; Bedrock cross-region inference profiles.
4. Quotas: tokens per minute and budget per key; soft alert at 80%, hard stop or downgrade at 100%.
5. Cost attribution: record input/output/cache tokens per request with team, app, model; price table versioned in the gateway; export to the data warehouse and a per-team dashboard. Bedrock application inference profiles plus cost allocation tags help reconcile with the AWS bill.
6. Observability: RED metrics per model and per team, time-to-first-token, tokens/s, provider error and throttle rates, traces with OpenTelemetry GenAI semantic conventions. Log prompts only where policy allows, with redaction.
7. Guardrails: PII redaction, prompt-injection and content filters (Bedrock Guardrails or equivalent), allow-list of models per data classification.
8. Caching: provider prompt caching surfaced to clients; optional exact-match response cache for deterministic calls.

Review checklist

• Can a team self-serve a key in minutes, with an owner and budget attached?
• Can finance see cost per team for last month without a manual spreadsheet?
• What happens when a region throttles? Tested?
• Is there a kill switch per key and per model?
• Are SLOs defined for the gateway itself (availability, added latency p99)?
• Are streaming responses handled end to end (timeouts, buffering at the load balancer)?

Output

An architecture description (and diagram in Mermaid if useful), the decisions with trade-offs, a phased rollout (pilot team → paved road → mandatory), and the SLOs and dashboards for the gateway.

MCP server review

Tool design

• Each tool does one job a user would name. Prefer create_ticket over a generic call_api.
• Names are verbs in snake_case; descriptions say when to use the tool and when not to, in one or two sentences.
• Input schemas are strict: enums instead of free strings, required fields marked, sensible limits on page sizes.
• Outputs are compact and model-friendly: return the fields an agent needs, paginate, avoid dumping raw API payloads.
• Errors explain how to fix the call ("project_id not found; list projects with list_projects").
• Separate read and write tools so clients can allow reads and require approval for writes.

Security

• Auth: OAuth 2.1 for remote servers, acting on behalf of the user; never a shared admin token for everyone.
• Least privilege: scopes per tool; destructive tools behind confirmation or not exposed.
• Treat everything the tools return as untrusted data: content from tickets, emails or web pages can carry prompt injection. Don't let tool output trigger other write tools without the user.
• No secrets in tool results or logs. Validate and sanitise inputs that reach shells, SQL or URLs (SSRF).
• Pin and review dependencies; publish who owns the server.

Operability

• Health endpoint, structured logs with a request id, metrics per tool (calls, errors, latency).
• Rate limits per user; timeouts on every upstream call.
• Versioned releases and a changelog; breaking schema changes get a new tool name.

Gateway / catalog readiness

When the server goes behind an MCP gateway or into an internal catalog, check: owner and support channel, data classification, which tools are write/destructive, required scopes, SLO, and an example prompt that exercises each tool.

Output

A table of findings by severity (blocker / should fix / nice to have) with the concrete change, then a go / no-go for onboarding.

Authoring skills and plugins for a marketplace

Skill structure

plugins/<plugin>/
  .claude-plugin/plugin.json
  skills/<skill-name>/SKILL.md
  skills/<skill-name>/reference.md   # optional, loaded only when needed
  skills/<skill-name>/scripts/       # optional helpers

SKILL.md starts with frontmatter:

---
name: <kebab-case, matches the folder>
description: <what it does>. Use when <the situations and words that should trigger it>.
---

The description decides everything

Claude sees only the description until the skill is invoked. Write it as: capability + "Use when" + the concrete triggers users actually say (product names, error messages, verbs). Avoid vague words like "helps with". Keep it under ~250 characters.

The body

• Lead with the goal and any hard rules (e.g. read-only, ask before destructive actions).
• Give a procedure the model can follow, branching on what it finds.
• Include the exact commands, queries or templates the team uses. That's the knowledge Claude doesn't have.
• End with what the output must look like.
• Move long reference material into separate files and link them, so the main file stays short.
• Don't restate things Claude already knows well; spend the words on company-specific context.

Plugin and marketplace

• plugin.json: name, description, author, keywords. Omit version to let every commit be a new version, or set it and bump it on each release.
• Add an entry to .claude-plugin/marketplace.json with name and source: "./plugins/<plugin>".
• Validate with claude plugin validate . before pushing.

Review checklist

1. Would the description trigger on the three most common ways people ask for this?
2. Does it avoid triggering on unrelated requests?
3. Can a new engineer follow the body without asking anyone?
4. Are destructive steps gated?
5. Is there an owner to maintain it?