Representative engagement This case study describes a typical engagement of this kind rather than a specific client. Figures are representative of the work, not a guarantee of results.
AI cost audit · Software
AI model spend cut by 54% with no measurable loss in answer quality
A 40-person legal-tech company's model bill had grown to $31,000 a month and was eating the margin on its AI contract-review product. One week of measurement produced a savings plan; three weeks of implementation delivered it.
Results
Measured over the first full month after implementation.
The problem
The product reviewed contracts with a large language model and had grown fast. Every request went to the most capable model with the full document in context, and the bill had tripled in six months.
Nobody could say what a single contract review cost, or which customers were unprofitable. Finance had asked for a forecast and the engineering team had no instrumentation to produce one.
Previous attempts to switch to a cheaper model had been abandoned after a customer complained, because there was no way to measure quality before and after.
What we did
One week of measurement, then implementation by the client's engineers with our support.
Days 1 to 2: attribution
Request traces and invoices were joined to attribute every dollar to a feature, a customer and a request type. Two features accounted for 70% of spend; 12 customers accounted for half of it.
Days 3 to 4: evaluation and modelling
A 240-example evaluation set was built from past reviews with lawyer-verified answers. Each savings lever was tested against it to estimate saving and quality effect.
Day 5: plan
A ranked plan: route clause classification to a small model, cache the shared instruction prefix, cut retrieved context from 20 chunks to 5, cap agent retries, and batch overnight re-processing. Dashboard and 12-month forecast delivered.
Implementation
The client's engineers implemented the plan over three weeks, running the evaluation set on every change. Cost per review fell from $4.10 to $1.85 with no regressions.
Stack
What made the difference
Attribution changed the conversation from 'AI is expensive' to 'these two features and these twelve customers are expensive', which is a problem a product team can act on.
The evaluation set was the enabler. Once quality was a number, the team made in a week the model changes it had been afraid to make for six months.
Routing and caching delivered most of the saving; retrieval tuning improved answer quality as well as cost.