Skip to main content

Code walkthrough

How a public meeting becomes a decision you can search, step by step in run order. Each step says what it does, links the code, and gives the command, what it writes and how to check it. Most data steps are package CLIs (python -m <package>.<module>); scripts/ is legacy and frozen (cleanup roadmap).

The short version: find governments and their sites → collect meetings and reference data → AI reads the transcripts → dbt cleans, joins and scores → copy the serving tables to Neon → the API serves them → the app shows them.

The steps in run order​

#StepCommand (from the repo root)Writes
1Find governments and their websitespython -m ingestion.census.municipalitiesbronze jurisdictions
2Collect meetingspython -m ingestion.youtube.eventsbronze meeting videos
3Load reference datapython -m ingestion.irs.bmfbronze nonprofits, Census, bills
4AI reads the transcriptspython -m llm.gemini.batch_backlog submit --state ALbronze_events_analysis_ai
5Clean, join and model in dbtcd dbt_project && dbt buildstaging, intermediate, marts, public
6Match the same entity across sourcespython -m ingestion.mdm addressbronze clusters, then dbt
7Policy questions and argumentspython -m llm.policy_questions.run_pipelinepolicy_question, canonical_argument
8Score items for the story lenses(in step 5's dbt build)item_interestingness
9Publish to Neonpython -m hosting.neon.sync_public_to_neon --dry-runNeon public
10Serve it: the APIpython main.py serve --port 8001JSON at /api/...
11Show it: the web appcd web_app && npm run devpages at localhost:5173
12DeployGitHub Actions: Deploy to Hugging Face Spacescommunityone.com

Every loader in steps 1 to 3 follows one contract, DataSourcePipeline (core_lib/pipeline/base.py): extract() yields raw rows, a Pydantic schema validates them, and load_batch() writes them in batches, with metrics and rollback. Read it once and every loader reads the same way.

1. Find governments and their websites​

The list of places comes from the Census: Gazetteer places, counties and school districts. Discovery then finds each government's website, its meeting platform (Legistar, Granicus and others) and its YouTube channel.

  • Code: ingestion/census/ (municipalities.py, counties.py, gazetteer.py, shapefiles.py); scrapers/discovery/ (platform_detector.py, url_discovery_agent.py, youtube_channel_verification.py).
  • Writes: jurisdiction rows in bronze, with website and channel URLs as they're found.
  • Check: the jurisdiction mapping-quality page in the app (/jurisdiction-quality) shows which places have a website and a channel, and which don't.

2. Collect meetings​

Meetings arrive from many platforms, so each platform has an adapter that turns its records into one Meeting shape. YouTube is the main source of video and captions.

3. Load reference data​

The context around a decision: nonprofits from the IRS, demographics and finance from the Census, state bills from Open States, and more (one folder per source under ingestion/).

4. AI reads the transcripts​

Gemini reads each meeting transcript with a fixed prompt and returns structured JSON: the decisions made, how each member voted, who spoke, and the quote for each. Batch mode sends the backlog as one file, answered within a day at a lower price.

5. Clean, join and model in dbt​

Every SQL transformation is dbt, never Python. Four layers: bronze (as landed) → staging (typed and renamed, years cast to integers) → intermediate (joined and enriched) → marts (what the app reads). Today that is 16 bronze, 71 staging, 55 intermediate and 102 mart models. The marts the API serves are published as views in public, each with a declared primary key and foreign keys.

  • Code: dbt_project/models/. One decision, as an example: marts/event_decision.sql reads bronze_events_analysis_ai and bronze_decisions_from_ai and joins them to event_meeting.
  • Run: cd dbt_project && dbt build (models and tests). Before changing a model, list what depends on it: dbt ls --select <model>+.
  • Check: dbt test; the contracts in data contracts; the diagrams in data model.

6. Match the same entity across sources​

The same person, address or nonprofit shows up in several sources under slightly different names. dbt builds the pooled inputs, Splink predicts which records are the same entity and clusters them, and dbt serves the result.

7. Policy questions and arguments​

Turns one-off votes into questions that recur across places. Decisions are embedded and clustered into place-neutral policy questions, and the arguments made for and against are clustered into a canonical argument library (the Key Point Analysis method), each tagged with a frame from the Policy Frames Codebook.

  • Code: llm/policy_questions/ (embed_instances.py, cluster_questions.py, cluster_arguments.py, labeler.py, run_pipeline.py).
  • Run: python -m llm.policy_questions.run_pipeline (no AI calls, for checking); add --use-llm for a real labelled run, which is billed.
  • Writes: policy_question, canonical_argument, question_instance, instance_argument marts.
  • Serves: /api/policy-question (policy_questions.py).

8. Score items for the story lenses​

Each decision and meeting item gets signals (split votes, money moved, unusual patterns, speed, upcoming dates), normalised and combined into a score. The five lenses on the home page (Contested, Money Moves, Raised Eyebrows, Moving Fast, Watch Next) are views over that score.

9. Publish to Neon​

The warehouse runs locally (Postgres on localhost:5433). Production reads Neon, a hosted Postgres. This step copies the built public serving objects to Neon as plain tables; it doesn't run dbt there.

  • Code: hosting/neon/sync_public_to_neon.py.
  • Run: python -m hosting.neon.sync_public_to_neon --dry-run first, then without it (prod) or with --target dev. --only <table> ... copies a subset.
  • Needs: NEON_DATABASE_URL (prod) or NEON_DATABASE_URL_DEV in .env.
  • Read: Neon deployment.

10. Serve it: the API​

FastAPI, one router per feature under api/routes/. Routes read the public schema and do no transformation. The database is the first of NEON_DATABASE_URL_DEV, NEON_DATABASE_URL, DATABASE_URL that is set (api/database.py).

RouteFileReads
/api/searchsearch.py, queries in search_postgres.pysearch tables; ts_headline builds the highlighted passage for each result
/api/decision/{id}decisions.pyevent_decision
/api/lenseslenses.pyitem_interestingness
/api/policy-questionpolicy_questions.pypolicy_question and its arguments
/api/money-and-talkmoney_and_talk.pytopic_money_and_talk
/api/statsstats_neon.pyjurisdiction_state_aggregate
  • Start: api/main.py registers the routers; ./start-all.sh runs it on port 8001 with the app and docs.
  • Check: http://localhost:8001/docs lists every route. The app's TypeScript types are generated from the OpenAPI schema (web_app/src/lib/api.types.ts), and CI checks they're current.

11. Show it: the web app​

React, Vite and TypeScript. Two rules shape every page that filters data: the scope label always names the filter actually applied, carried in the URL (?state=&city=), and a result found by a search term shows the passage that matched, highlighted.

12. Deploy​

Production is one Docker container on Hugging Face Spaces. nginx serves the built app at / and the docs at /docs/, and passes /api/ to uvicorn on port 8000; supervisord keeps both running.

Where to change things​

To changeStart in
A new data sourcea folder under packages/ingestion/src/ingestion/, on the DataSourcePipeline contract
What the AI extracts from a transcriptprompts/policy_analysis_part_1.md, then step 5's bronze models
A number or list the app showsits mart in dbt_project/models/marts/, never the API
A new API endpointa router in api/routes/, registered in api/main.py; regenerate api.types.ts
A pageweb_app/src/pages/, following the scope-label and matched-passage rules in CLAUDE.md
The rules themselvesCLAUDE.md and AGENTS.md