AI Architecture Audit
A fixed-scope audit that measures where your LLM stack leaks money and accuracy, run on your real usage data, ending in a prioritized fix plan your team can execute.
- Timeline
- 2-3 weeks, fixed scope
- Team
- AI architect, MLE; security reviewer for agent-tooling findings
- Typical stack
- Provider usage exports (OpenAI, Anthropic, Google), token-log analysis, retrieval replay against your corpus, latency profiling. Findings map to your stack: LangGraph/Temporal, pgvector/Pinecone/Weaviate, and your routing layer as applicable.
What you get
- Measured cost-leakage analysis from your real usage exports (billing, token logs), not modeled assumptions
- Retrieval and chunking review: over-retrieval, K and hop settings, context-window load, embedding fit
- Model-routing review: which calls belong on cheaper models, caching opportunities, batch candidates
- Failure-mode assessment: grounding risk, drift exposure, prompt-injection surface for agent tooling
- Prioritized fix plan with cost and effort sizing for each finding
- Executive readout: a written brief your leadership can act on without us in the room
Outcomes
- Your actual monthly leakage number, measured against your own bill
- A ranked fix list your team can execute with or without us
- Estimate-versus-measured deltas reported openly
- A clear call: fix, rebuild, or leave it alone
What we need from you
- Usage and billing exports (read-only) from your model providers
- Architecture description: models, vector store, orchestration, chunking config
- Eval results or a sample of production traces, if available
- One technical owner for a weekly working session
Proof points
- The free diagnostic that fronts this audit uses published list prices and stated assumptions; every number is labeled as an estimate
- Method example: a month of usage at $8.8k with a 22.6:1 input-to-output token ratio is a textbook over-retrieval leak, roughly $3.4k/mo recoverable
- Deliverables are yours under work-for-hire: findings, fix plan, and readout
Frequently asked questions
The diagnostic is a modeled estimate: you describe your stack and deterministic formulas project leakage from published list prices and stated assumptions. The audit measures the real thing: we analyze your actual usage exports and traces, so the numbers are yours, not projections. The estimate tells you whether to look. The audit tells you what to fix and in what order.
That is one of the specific questions the audit answers, and the finding is usually more nuanced than teams fear. Most companies overestimate the data needed for retrieval and workflow use cases and underestimate what forecasting needs. The audit maps the use cases you want against the data you have: what works today, what needs pipeline work first, and what to stop collecting because nothing will ever use it.
Partly, and you should start there: the free 3-minute diagnostic is exactly that. What an external audit adds is a pattern library (we have seen how these decisions play out across many estates) and independence: internal assessments inherit internal politics, and the systems most in need of scrutiny usually have the most invested defenders. If the diagnostic already tells you what you need, take the roadmap and run.
Yes, that is the point of the audit. Read-only exports under NDA; nothing connects to your systems. You export, we analyze. If you cannot share usage data yet, start with the free diagnostic, which needs nothing but a description of your stack.
The most common leak is over-retrieval: sending far more context than the answer needs, which shows up as a high input-to-output token ratio. After that: every call routed to the most expensive model, no caching on repeated queries, and chunking that fights the retriever. Not every stack has all four. Occasionally an architecture comes back healthy, and we say so.
You get the fix plan either way, written so any competent team can execute it, yours or ours. If you want us to implement, the audit findings scope the build engagement directly. No obligation: the audit is a deliverable, not a sales gate.