Databricks Free Edition
What it's for: processing that shouldn't depend on the maintainer's laptop. The free target of the
agent-analysis-sync bundle runs the transcript analysis (dbt with ai_query) on serverless compute, and creates the
landing volume the nightly crawl and the Colab GPU steps write to
(#293). It is set up the same way as the Health
Transformation Lab's dev workspace: a service principal, a GitHub environment, and a deploy on every merge.
Limits to plan around
- Serverless only; small compute with a daily usage limit. Over it, compute stops for the rest of the day (data stays).
- One SQL warehouse (2X-Small), at most 5 concurrent job tasks per account, one active pipeline per type.
- Outbound internet is limited to trusted domains unless you verify with LinkedIn. That's why crawling runs on GitHub Actions, not here.
- Not for commercial use. Check the terms fit CommunityOne's use.
- One workspace per account. If the Health Transformation Lab already uses your Free Edition workspace, this bundle can
share it (its own catalog,
open_navigator), and the two projects share the daily limit. - Not yet checked on Free Edition: whether the
databricks-gpt-oss-120bpay-per-token endpoint is available toai_querythere. The first run says; if not, setprimary_modelto an endpoint that is.
Source: Free Edition limitations.
Connect it from a phone (about 5 minutes)
The deploy also signs in with a personal access token, so the workspace can be connected without a terminal. Replace the token with the service principal (below) when you're at a computer.
- In the Free Edition workspace, open your profile (top right) → Settings → Developer → Access tokens → Generate new token. Comment "open-navigator CI", lifetime 30 days. Copy it.
- In GitHub: repository Settings → Environments → New environment named
free. Add the variableDATABRICKS_HOST(the workspace URL, e.g.https://dbc-xxxx.cloud.databricks.com) and the secretDATABRICKS_TOKEN(the token). Never paste the token anywhere else. - Actions → Deploy to Databricks Free Edition → Run workflow. It checks the connection, creates the
open_navigatorcatalog once, then validates and deploys.
The token acts as you, with your rights in the workspace; the service principal has only what the job needs. Delete the token in the workspace when the service principal is in place.
One-time setup (about 15 minutes)
- Sign up at databricks.com/learn/free-edition (or use the workspace you already have), and choose Verify with LinkedIn in the workspace.
- Log in from a terminal:
databricks auth login --host https://<workspace>. - Run
apps/agent-analysis-sync/setup/create_service_principal.sh free. It creates the service principal and admin group, theopen_navigatorcatalog, and the grants, then prints the next commands. - Make the principal's OAuth secret (90 days) and put it straight into the GitHub environment
free, with the variablesDATABRICKS_HOSTandDATABRICKS_CLIENT_ID, exactly as the script prints. Never paste the secret into a file or chat. - Run Actions → Deploy to Databricks Free Edition → Run workflow. It validates and deploys; tick run_analysis to also run one batch.
- Load transcripts once from a machine with the local warehouse, pointing the loader at the Free Edition workspace
(its defaults are the old Azure workspace):
python -m ingestion.databricks.load_transcripts_to_uc --profile <free profile> --warehouse-id <starter warehouse id> --catalog open_navigator --limit 25.
What the deploy creates
| Object | Name |
|---|---|
| Analysis schema | open_navigator.open_navigator_analysis |
| Landing schema and volume | open_navigator.open_navigator_landing.crawl |
| Job (schedule paused) | [free] agent-analysis-sync |
Batches are 25 transcripts (analysis_batch_size), small enough to stay under the daily limit. Raise it once a run
shows how much quota a batch uses.
Credential
| Credential | Where it's made | Scope | Goes to | Lifetime |
|---|---|---|---|---|
| Personal access token (phone setup only) | Workspace → Settings → Developer → Access tokens | your user's rights | GitHub environment free, secret DATABRICKS_TOKEN | 30 days; delete once the service principal is set up |
open_navigator_free_sp OAuth secret | Databricks CLI: service-principal-secrets-proxy create (printed by the setup script) | the service principal: workspace access, SQL access, ALL PRIVILEGES on open_navigator | GitHub environment free, secret DATABRICKS_CLIENT_SECRET | 90 days; renew 14 days before |