Chapter 2: Build a useful cost baseline

Overview#

Say one model handles support classification, document extraction and an internal research agent. The provider dashboard shows the model cost $2,600 last month, but probably not which feature spent it or why.

Each feature can have a different problem: oversized prompts in one, repeated paid searches in another, retries in the third. Before switching models, attribute spending to the operation responsible for it.

Practical#

Give each operation a task_id and attach it to every model event, tool invocation and retry. Store the final task outcome separately from individual request outcomes.

A minimal event record needs task_id, operation, event_type, provider, model, input/output/cached token counts when available, estimated or invoiced cost, attempt number and latency. Add a timestamp and the version of the price table used to compute cost. Metadata is usually enough for cost analysis, so don’t store sensitive prompts just to build a dashboard.

The report below counts accepted tasks, so define accepted before you count. For extraction, a task might need to pass three checks: the response matches the schema (shape), dates, currencies and totals satisfy business rules (consistency), and the values agree with the source document (correctness). Each operation needs its own definition, because a support reply or an agent run succeeds in different ways. Record every task as accepted, rejected, incomplete or pending_review, so failed and unfinished work stays in the data with its cost.

From that, produce a short report:

Illustrative data. The research agent has the highest unit cost; extraction has the biggest absolute bill. Which to optimize depends on task value, unnecessary work and the feasible saving.

Instead of asking which model is expensive, look for the operation that pays most for retries, keeps resending unchanged context, calls the most external tools or shows the widest spread between ordinary and costly runs.

What you pay is computed from the provider’s records, so treat them as the source of truth. Pull usage and cost from the provider usage and cost APIs or billing exports, reconcile them with your event totals every month and investigate any gap. Attribution is easier when spend is separated at the source: use separate API keys or metadata tags per feature, team or environment, so test traffic doesn’t land on a production feature’s bill. If you already run an LLM observability tool, or emit traces that follow OpenTelemetry’s GenAI semantic conventions, send the same task_id there instead of building a second pipeline.

Then give the numbers owners. Set budgets and alerts per team. Use showback, where each team sees its own spend, or chargeback, where that spend is billed to its cost center. Keep a simple monthly forecast per operation: expected volume multiplied by the current cost per accepted task. When actual spend drifts from it, check whether volume or unit cost changed.

Ledgerline. The table above is Ledgerline’s first report. Adding a task_id to every call, search and retry took two days. The result surprised the team: the research agent produces under 3% of accepted tasks (2,000 of 72,000) but 31% of the bill ($800 of $2,600). Most chapters from here on change one of these three lines.

Deep Dive#

A simple PostgreSQL event table can support this analysis:

CREATE TABLE ai_cost_events (
  task_id TEXT NOT NULL,
  operation TEXT NOT NULL,
  event_type TEXT NOT NULL,
  model TEXT,
  input_tokens BIGINT DEFAULT 0,
  output_tokens BIGINT DEFAULT 0,
  cached_tokens BIGINT DEFAULT 0,
  cost_usd NUMERIC(18, 8) NOT NULL,
  attempt INTEGER DEFAULT 1,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Aggregate events by task first, then join to a task-outcomes table. That way one operation with eight calls doesn’t count as eight completed tasks. Record excluded fixed costs and incomplete events explicitly, so an incomplete bill is never presented as total cost.

If your task mix changes between months, the average unit cost can move with no change in model efficiency. Compare like with like, or break the metric down by task type and complexity.

Define acceptance before comparing model costs#

The word accepted needs an operational meaning. Otherwise the denominator becomes a matter of opinion, and the cheapest implementation will often appear to win because it drops the difficult cases.

The three extraction checks need different tools. A schema validator handles shape, and application code can handle parts of consistency. Correctness may need source-grounded comparison or review, because a response can pass the first two checks and still be wrong.

For a documentation assistant, an accepted response might need to answer the question, cite a current authorized source and avoid unsupported claims. For a coding agent, acceptance might require the requested change to be present, no unrelated modifications and the defined acceptance conditions to be satisfied. A classification task may have a deterministic ground-truth label; an open-ended research task may need graded review and explicit unresolved states.

Disclose which of the four task states enter your denominator. For a monthly report, define the cohort and cutoff: tasks created this month, or tasks completed this month? If you mix the two, cost and acceptance rates can disagree only because some runs cross month boundaries.

Report the acceptance rate next to the cost per accepted task. Either number alone can mislead:

Illustrative scenario. The two operations produce different kinds of outcome, so their unit costs can’t be ranked against each other directly.

A calculation that joins the events to the outcomes#

This PostgreSQL query makes the denominator explicit. It takes the tasks started in the month as the cohort, so rejected, incomplete and still-pending work keeps its cost in the numerator instead of disappearing from the report. It assumes one row per task with its current state, and that the event table includes all directly attributable variable costs. Shared fixed capacity and human review need their own allocations before this becomes a full TCO report.

WITH task_cost AS (
  SELECT task_id, SUM(cost_usd) AS cost_usd
  FROM ai_cost_events
  GROUP BY task_id
), cohort AS (
  SELECT t.task_id, t.operation, t.status,
         COALESCE(c.cost_usd, 0) AS cost_usd
  FROM ai_tasks AS t
  LEFT JOIN task_cost AS c USING (task_id)
  WHERE t.created_at >= DATE '2026-10-01'
    AND t.created_at < DATE '2026-11-01'
)
SELECT operation,
       COUNT(*) AS started_tasks,
       COUNT(*) FILTER (WHERE status = 'accepted') AS accepted_tasks,
       COUNT(*) FILTER (WHERE status = 'pending_review') AS pending_tasks,
       SUM(cost_usd) AS variable_cost_usd,
       SUM(cost_usd) / NULLIF(
         COUNT(*) FILTER (WHERE status = 'accepted'), 0
       ) AS cost_per_accepted_task
FROM cohort
GROUP BY operation;

The query requires an ai_tasks table with task_id, operation, created_at and status (accepted, rejected, incomplete or pending_review). Treat the month as provisional while pending_tasks is still significant: each pending task that ends up accepted lowers the cost per accepted task. In real accounting, keep usage events with a zero or unknown cost distinct from events confirmed as free. Reconcile the sums against invoices, because pricing calculations can miss cached-token discounts, tool charges or delayed usage records.

Document the acceptance definition and the accounting exclusions next to the report, so another engineer can reproduce the numbers.

The table definitions and the monthly query are also in a SQL file you can download, with the task table added.