Skip to main content

Databricks Free Edition

What it's for: processing that shouldn't depend on the maintainer's laptop. The free target of the agent-analysis-sync bundle runs the transcript analysis (dbt with ai_query) on serverless compute, and creates the landing volume the nightly crawl and the Colab GPU steps write to (#293). It is set up the same way as the Health Transformation Lab's dev workspace: a service principal, a GitHub environment, and a deploy on every merge.

Limits to plan around
  • Serverless only; small compute with a daily usage limit. Over it, compute stops for the rest of the day (data stays).
  • One SQL warehouse (2X-Small), at most 5 concurrent job tasks per account, one active pipeline per type.
  • Outbound internet is limited to trusted domains unless you verify with LinkedIn. That's why crawling runs on GitHub Actions, not here.
  • Not for commercial use. Check the terms fit CommunityOne's use.
  • One workspace per account. If the Health Transformation Lab already uses your Free Edition workspace, this bundle can share it (its own catalog, open_navigator), and the two projects share the daily limit.
  • Not yet checked on Free Edition: whether the databricks-gpt-oss-120b pay-per-token endpoint is available to ai_query there. The first run says; if not, set primary_model to an endpoint that is.

Source: Free Edition limitations.

Connect it from a phone (about 5 minutes)

The deploy also signs in with a personal access token, so the workspace can be connected without a terminal. Replace the token with the service principal (below) when you're at a computer.

  1. In the Free Edition workspace, open your profile (top right) → Settings → Developer → Access tokens → Generate new token. Comment "open-navigator CI", lifetime 30 days. Copy it.
  2. In GitHub: repository Settings → Environments → New environment named free. Add the variable DATABRICKS_HOST (the workspace URL, e.g. https://dbc-xxxx.cloud.databricks.com) and the secret DATABRICKS_TOKEN (the token). Never paste the token anywhere else.
  3. Actions → Deploy to Databricks Free Edition → Run workflow. It checks the connection, creates the open_navigator catalog once, then validates and deploys.

The token acts as you, with your rights in the workspace; the service principal has only what the job needs. Delete the token in the workspace when the service principal is in place.

One-time setup (about 15 minutes)
  1. Sign up at databricks.com/learn/free-edition (or use the workspace you already have), and choose Verify with LinkedIn in the workspace.
  2. Log in from a terminal: databricks auth login --host https://<workspace>.
  3. Run apps/agent-analysis-sync/setup/create_service_principal.sh free. It creates the service principal and admin group, the open_navigator catalog, and the grants, then prints the next commands.
  4. Make the principal's OAuth secret (90 days) and put it straight into the GitHub environment free, with the variables DATABRICKS_HOST and DATABRICKS_CLIENT_ID, exactly as the script prints. Never paste the secret into a file or chat.
  5. Run Actions → Deploy to Databricks Free Edition → Run workflow. It validates and deploys; tick run_analysis to also run one batch.
  6. Load transcripts once from a machine with the local warehouse, pointing the loader at the Free Edition workspace (its defaults are the old Azure workspace): python -m ingestion.databricks.load_transcripts_to_uc --profile <free profile> --warehouse-id <starter warehouse id> --catalog open_navigator --limit 25.
What the deploy creates
ObjectName
Analysis schemaopen_navigator.open_navigator_analysis
Landing schema and volumeopen_navigator.open_navigator_landing.crawl
Job (schedule paused)[free] agent-analysis-sync

Batches are 25 transcripts (analysis_batch_size), small enough to stay under the daily limit. Raise it once a run shows how much quota a batch uses.

Credential
CredentialWhere it's madeScopeGoes toLifetime
Personal access token (phone setup only)Workspace → Settings → Developer → Access tokensyour user's rightsGitHub environment free, secret DATABRICKS_TOKEN30 days; delete once the service principal is set up
open_navigator_free_sp OAuth secretDatabricks CLI: service-principal-secrets-proxy create (printed by the setup script)the service principal: workspace access, SQL access, ALL PRIVILEGES on open_navigatorGitHub environment free, secret DATABRICKS_CLIENT_SECRET90 days; renew 14 days before