Most AI ROI discussions fail because teams report activity metrics ("number of prompts," "agent usage," "tokens consumed") instead of operating outcomes. Executive sponsors do not approve scale based on excitement. They approve based on dependable quality, measurable time impact, and risk-adjusted cost.
This playbook gives a simple, repeatable ROI model you can use in governance and steering meetings. It ties directly to outcome framing in From Prompts to Business Outcomes and to case evidence patterns used in the Northline series, including Northline Part 2: Scaling Eval Coverage and Case Study: Vibe Prompting to Structured Workflow.
The three ROI drivers
Track these three as a bundle. Alone, each one can mislead.
| Driver | What it captures | Typical pitfall |
|---|---|---|
| Pass rate | Quality and policy compliance at release and in production | Measuring only happy-path demos |
| Time saved | Throughput and labor impact on real workflow steps | Ignoring review/rework time |
| Incident cost | Financial and trust impact of failures, escalations, or policy misses | Treating incidents as "rare noise" |
A workflow with fast output but poor pass rate is not ROI-positive. A workflow with high pass rate but no throughput impact may still be worth running for risk reduction, but it should be funded as control, not productivity.
Core formulas (v1)
Use these starter formulas in your scorecard:
1) Quality-adjusted throughput gain
Quality-adjusted gain = (baseline volume x baseline pass rate) -> (new volume x new pass rate)
In practice, teams often hold volume constant first and measure effective good outputs per week.
2) Net time savings
Net time savings = (baseline task minutes - new task minutes) x task count - oversight minutes
Include reviewer time, exception handling, and rework. Do not subtract only "model generation time."
3) Risk-adjusted value
Risk-adjusted value = gross productivity value - (incident count x avg incident cost)
Incident cost should include triage time, customer impact credits, compliance work, and opportunity cost from paused releases.
Scorecard template for monthly review
Monthly ROI review fails when metrics float without owners. The template below assigns one accountable name per row so "shared responsibility" does not mask drift. Baseline values should be frozen in the workflow changelog when automation promotes from pilot; without a frozen baseline, sponsors debate whether gains are real or narrative. Northline's finance partner co-signed the scorecard for support-reply-v3, which made incident cost estimates defensible in steering meetings instead of hand-waved.
| Metric | Baseline | Current | Delta | Owner |
|---|---|---|---|---|
| Eval pass rate (held-out) | Process owner | |||
| Production acceptance rate | Ops lead | |||
| Median cycle time per case | Process owner | |||
| Human review minutes per case | Team lead | |||
| Incidents per 1,000 runs | Governance lead | |||
| Average incident cost | Finance + Ops | |||
| Net monthly ROI estimate | Sponsor |
Use one owner per metric to avoid "shared responsibility" drift.
Pass rate: what to count
Pass rate is the quality anchor of ROI—without it, time savings may just be faster production of wrong outputs. Split eval pass rate (controlled release gate) from production acceptance rate (live rewrite burden) so you detect stale test sets before expanding scope. When eval is high but acceptance drops, update cases and hold promotion; do not explain the gap away as "users need training." Northline tracks both layers on the same monthly scorecard row so sponsors see quality and throughput together.
Treat pass rate in two layers:
- Eval pass rate: controlled release gate across fixed test cases.
- Production acceptance rate: percentage of live outputs accepted without major rewrite.
If eval is high but production acceptance drops, your test set is stale. Update cases before expanding automation scope. This is the same discipline used in AI Workflow Eval Checklist and Evaluation Hooks for AI Workflows.
Time saved: remove optimism bias
First-draft speed is the most seductive and misleading ROI metric. Agents that generate text in seconds but add eight minutes of correction and policy rework often increase net cycle time. Measure end-to-end medians across intake, generation, review, policy checks, and final action—then subtract oversight minutes explicitly. Northline's support-assist pilot looked fast on draft-only timers until they included reviewer edits; the honest full-cycle comparison is what justified phase-two rollout to executives.
Teams overestimate time savings when they measure only first draft speed. Measure end-to-end cycle:
- Intake and context assembly
- Draft generation
- Review and edits
- Policy checks
- Final action (send, publish, or commit)
A workflow that drafts in 30 seconds but adds 8 minutes of correction is not faster. Compare full-cycle medians before and after adoption.
Incident cost: include hidden costs
Incident cost turns ROI from a productivity story into a risk-adjusted business case. Teams undercount triage labor, customer remediation, compliance review, and delayed releases—then wonder why sponsors hesitate to scale after a near-miss. Build incident cost bands with finance and operations using historical escalations, not intuition. Northline estimated average incident cost at $280 per event for support-assist workflows; that figure made reliability improvements in pass rate and incident rate legible to non-technical steering members.
Incident cost is often underreported. Include at least:
- Response and triage labor
- Rework and rollback activity
- Customer remediation (credits, escalations)
- Compliance/legal review effort
- Opportunity cost from delayed promotions
This aligns with risk discipline in AI Risk Review Cadence and evidence expectations in Audit Trails for AI Workflows.
Northline-style worked example
Northline B2B tracked a support-assist workflow over one month:
- Baseline pass rate: 84%
- New pass rate: 93%
- Cases/month: 1,200
- Baseline cycle time: 14 minutes
- New cycle time: 10 minutes
- Oversight overhead: 1.5 minutes/case
- Incident rate dropped from 9 to 5 per 1,000 runs
- Estimated average incident cost: $280
Outcome:
- Effective high-quality outputs increased by roughly 12%.
- Net time impact remained positive after oversight inclusion.
- Incident cost reduction materially improved sponsor confidence for phase-two rollout.
The key lesson was not "AI is cheaper." It was "quality and risk controls made productivity gains credible."
Executive-ready reporting pattern
Sponsor decks fail when they lead with speed and bury quality. Present quality first, time second, risk third, then the decision—expand, hold, or rollback. That sequence prevents teams from pushing low-quality gains into production to satisfy quarterly narratives. Northline's steering template mirrors this order; when pass rate dipped after a retrieval refresh, the hold decision was obvious before anyone debated token costs.
Present ROI in this order:
- Quality first: pass rate trend and failure categories.
- Time second: net cycle-time impact with oversight included.
- Risk third: incidents and estimated avoided cost.
- Decision: expand, hold, or rollback automation scope.
This sequence prevents teams from pushing low-quality gains into production just to show speed.
Monday checklist
- Define baseline values for pass rate, cycle time, and incident cost.
- Assign explicit metric owners in one monthly scorecard.
- Add oversight time and rework time to "time saved" calculations.
- Estimate incident cost bands with finance and operations, not intuition.
- Use the next governance review to decide scope based on risk-adjusted ROI.
ROI is not a dashboard decoration. It is a release-governance tool. When pass rate, time, and incident cost move together in the right direction, you have evidence to scale. When they diverge, you have evidence to slow down before damage compounds.