AI Architecture Audit
A fixed-scope audit that measures where your LLM stack leaks money and accuracy, run on your real usage data, ending in a prioritized fix plan your team can execute.
- Timeline
- 2-3 weeks, fixed scope
- Team
- AI architect, MLE; security reviewer for agent-tooling findings
- Typical stack
- Provider usage exports (OpenAI, Anthropic, Google), token-log analysis, retrieval replay against your corpus, latency profiling. Findings map to your stack: LangGraph/Temporal, pgvector/Pinecone/Weaviate, and your routing layer as applicable.
What you get
- Measured cost-leakage analysis from your real usage exports (billing, token logs), not modeled assumptions
- Retrieval and chunking review: over-retrieval, K and hop settings, context-window load, embedding fit
- Model-routing review: which calls belong on cheaper models, caching opportunities, batch candidates
- Failure-mode assessment: grounding risk, drift exposure, prompt-injection surface for agent tooling
- Prioritized fix plan with cost and effort sizing for each finding
- Executive readout: a written brief your leadership can act on without us in the room
Outcomes
- Your actual monthly leakage number, measured against your own bill
- A ranked fix list your team can execute with or without us
- Estimate-versus-measured deltas reported openly
- A clear call: fix, rebuild, or leave it alone
What we need from you
- Usage and billing exports (read-only) from your model providers
- Architecture description: models, vector store, orchestration, chunking config
- Eval results or a sample of production traces, if available
- One technical owner for a weekly working session
Proof points
- The free diagnostic that fronts this audit uses published list prices and stated assumptions; every number is labeled as an estimate
- Method example: a month of usage at $8.8k with a 22.6:1 input-to-output token ratio is a textbook over-retrieval leak, roughly $3.4k/mo recoverable
- Deliverables are yours under work-for-hire: findings, fix plan, and readout