Skip to main content

LLM and GPU usage tracking

Every model call and every GPU step writes one usage row, so a model, prompt or batching choice can be compared on measured numbers. The work is logged in #262. The code is packages/llm/src/llm/usage/.

What is recorded​

RecordOne row perWritten byWhat it holds
Model callcall, after its retriescall_gemini_text (Gemini and DeepSeek), the local Gemma backendinput, cached input, output and thinking tokens; latency including retries; attempts; outcome
GPU stepstep wrapped in gpu_run(...)WhisperX diarization, a local Gemma generationwall time, GPU count and name, GPU-hours, mean utilization, peak memory, energy

A failed call is recorded too, because its latency and attempts are what the retries cost. A count the API does not report is stored as null. It is never estimated.

Rows are appended to JSON-lines files under LLM_USAGE_LOG_DIR (default data/logs/llm_usage/):

  • llm_calls_YYYYMMDD.jsonl
  • gpu_runs_YYYYMMDD.jsonl
  • gpu_samples/<gpu_run_id>.csv, the raw nvidia-smi samples for one step

The transcript pipeline also stores each Part 1 call's numbers in the meeting's *.meta.json, under usage.

Set LLM_USAGE_LOG=0 to turn recording off. A recording failure is logged as a warning and never stops a run.

Labelling calls​

A caller labels the calls it makes with llm_call_context. Only stage, video_id, jurisdiction_id and api_mode are kept, so a prompt or transcript cannot reach the log by accident.

from llm.usage.calls import llm_call_context

with llm_call_context(stage="part1", video_id=video_id, jurisdiction_id=jurisdiction_id):
result = call_gemini_text(...)
result.usage.thinking_token_count

A GPU step is wrapped the same way. On a machine with no nvidia-smi, or with device="cpu", it records nothing.

from llm.usage.gpu import gpu_run

with gpu_run("whisperx_diarization", device=args.device, video_id=video_id):
diarize_audio_whisperx(...)

Reading the numbers​

python -m llm.usage.report # by day, model and API mode
python -m llm.usage.report --since 2026-11-01 --by stage
python -m llm.usage.report --json

To query them in Postgres, load the logs. The loader applies migration 111_create_bronze_llm_usage.sql if the tables are missing, and a re-run adds only new rows.

python -m llm.usage.load_bronze --dry-run
python -m llm.usage.load_bronze
TableGrainPrimary key
bronze.bronze_llm_callone model callllm_call_id
bronze.bronze_gpu_runone GPU stepgpu_run_id

Cost​

Cost is not stored. The report multiplies the token counts and GPU-hours by the prices in packages/llm/src/llm/usage/prices.yaml, so a corrected price reprices all history.

A price is used only when it has a value, a source and a read_date. Anything else shows as a dash. The Gemini rates, interactive and batch, come from Google's Gemini API pricing page (paid tier, text rates, read November 14, 2026; #266). A rate can end: valid_until is the last day it applies, and a call after that day shows no cost until the new rate is added (Google doubles the 3.6 and 3.8 Flash rates on January 1, 2027; #333). Gemma 4 has no paid price (Google offers it on the free tier only). DeepSeek prices by time of day, so its entries (deepseek-flash, deepseek-v4-pro, and the legacy name deepseek-v4-flash) carry peak hours and an off_peak rate. A call is priced at the rate for its UTC time, and a call with no time of day shows no cost. Peak is 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays. DeepSeek also makes Chinese public holidays off-peak; those dates aren't recorded, so on them the estimate errs high (read November 14, 2026; #328). deepseek-chat, the name used before, is no longer listed, so its calls show no cost. Google has deprecated gemini-2.5-flash: from December 1, 2026 its calls are billed as gemini-3.5-flash-lite (#329). The GPU prices are estimates of what the GPU time would rent for on a cloud (a Colab or Kaggle GPU is paid in compute units or not at all).

Decision: the same method as health-transformation-lab-nlp​

The method follows the health-transformation-lab-nlp project, so the two projects' numbers can be read the same way: token counts taken from the API's own response, an nvidia-smi loop sampled to a CSV, GPU-hours as session time by GPU count, and a prices file where every price carries its source and the date it was read.

Two things differ, on purpose:

  • Where the rows go. That project puts token counts on OpenTelemetry spans. Here the pipelines are standalone package CLIs, which this repository logs with loguru (CLAUDE.md), so the rows go to JSON-lines files and a bronze table. OpenTelemetry stays for the FastAPI service.
  • Cost is derived, not stored. That project stores an estimated cost on each run. Here the report computes it, so no row holds a cost computed from a price that was later corrected.

The governance pipeline​

The governance (Colab) pipeline calls the Gemini API directly in gatekeeper_triage.py and governance_meeting_llm.py, through its own retry wrapper (governance/genai_quota_retry.py). Given a model, that wrapper writes the same usage row (#268). Each call is labelled by stage: gatekeeper, shield, or the demo that made it (demo1 to demo4, set by the demo runners in colab_demos.py), else governance.

In dbt​

bronze.bronze_llm_call and bronze.bronze_gpu_run are dbt sources (#268):

ModelGrainKey
stg_llm__call (view)one model callllm_call_id
stg_llm__gpu_run (view)one GPU stepgpu_run_id
llm_usage_daily (gold, contract enforced)one day (UTC), model and API mode(usage_date, model, api_mode)

Both staging views build empty until python -m llm.usage.load_bronze creates the bronze tables, so dbt build passes before any usage is loaded. llm_usage_daily has no cost column, for the reason above, and is not published to public.