Databricks Apps Deployment Guide

Overview

The Oral Health Policy Pulse application has been refactored as a React + FastAPI Databricks App, providing:

• 🎨 Modern React UI with TypeScript, Tailwind CSS, and interactive visualizations
• ⚡ FastAPI Backend serving both API and static frontend
• ☁️ Databricks Apps Deployment with Unity Catalog integration
• 🔒 Enterprise Security via Databricks secrets and authentication
• 📊 Real-time Analytics powered by Delta Lake and Model Serving

---

Architecture

┌─────────────────────────────────────────────────────────┐
│                   Databricks Workspace                   │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │         Databricks App (This Application)        │   │
│  │                                                  │   │
│  │  ┌──────────────┐      ┌──────────────────┐    │   │
│  │  │ React        │ ───▶ │ FastAPI Backend   │    │   │
│  │  │ Frontend     │      │                   │    │   │
│  │  │ (TypeScript) │      │ • REST API        │    │   │
│  │  │              │      │ • Static Serving  │    │   │
│  │  └──────────────┘      └──────────────────┘    │   │
│  │                               │                 │   │
│  └───────────────────────────────┼─────────────────┘   │
│                                  │                      │
│  ┌───────────────────────────────▼─────────────────┐   │
│  │          Unity Catalog & Delta Lake             │   │
│  │  • raw_documents                                │   │
│  │  • classified_documents                         │   │
│  │  • advocacy_opportunities                       │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │           Model Serving Endpoints                │   │
│  │  • policy-classifier-prod                       │   │
│  │  • sentiment-analyzer-prod                      │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
└─────────────────────────────────────────────────────────┘

---

Prerequisites

1. Databricks Workspace

• Databricks Runtime 14.3 LTS ML or higher
• Unity Catalog enabled
• Model Serving enabled

2. Local Development Tools

• Python 3.11+
• Node.js 20+
• Databricks CLI

3. API Keys

• OpenAI API key (for LLM-based classification)
• Anthropic API key (optional)

---

Local Development

Setup

# Clone and setup
cd open-navigator
./scripts/setup-local.sh

Run Development Server

Option 1: Separate Frontend + Backend (Hot Reload)

# Terminal 1 - Backend
source venv/bin/activate
uvicorn api.app:app --reload

# Terminal 2 - Frontend  
cd frontend
npm run dev

Visit: http://localhost:3000

Option 2: Production Mode (Serves Built Frontend)

# Build frontend first
cd frontend
npm run build
cd ..

# Run app
source venv/bin/activate
python scripts/test-app.py

Visit: http://localhost:8000

---

Deploying to Databricks Apps

Step 1: Configure Environment

# Set Databricks credentials
export DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
export DATABRICKS_TOKEN=dapi1234567890abcdef

# Set API keys
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

Step 2: Deploy

# One-command deployment
./scripts/deploy-databricks-app.sh

This script will:

1. ✅ Build React frontend (optimized production build)
2. ✅ Create Databricks secrets for credentials
3. ✅ Deploy app to Databricks Apps
4. ✅ Configure Model Serving endpoints
5. ✅ Provide access URL

Step 3: Access Your App

Once deployed, access at:

https://your-workspace.cloud.databricks.com/apps/open-navigator

---

Configuration

app.yaml

The main configuration file for Databricks Apps:

command:
  - "uvicorn"
  - "api.app:app"
  - "--host"
  - "0.0.0.0"
  - "--port"
  - "8000"

env:
  - name: DATABRICKS_HOST
    valueFrom:
      databricksSecret:
        key: host
        scope: oral-health-app
  - name: OPENAI_API_KEY
    valueFrom:
      databricksSecret:
        key: openai_key
        scope: oral-health-app

resources:
  - name: policy-classifier-endpoint
    modelServing:
      endpoint: policy-classifier-prod

port: 8000

Environment Variables

Variable	Description	Required
DATABRICKS_HOST	Workspace URL	Yes
DATABRICKS_TOKEN	Access token	Yes
DATABRICKS_WAREHOUSE_ID	SQL warehouse ID	Yes
OPENAI_API_KEY	OpenAI API key	Yes
ANTHROPIC_API_KEY	Anthropic API key	No
CATALOG_NAME	Unity Catalog name	Yes
SCHEMA_NAME	Schema name	Yes

---

Frontend Features

Dashboard

• 📊 Real-time statistics
• 📈 Topic distribution charts
• 📋 Recent opportunities list

Interactive Heatmap

• 🗺️ Geographic visualization
• 🎯 Filterable by state, topic, urgency
• 💡 Click markers for details

Documents Browser

• 🔍 Full-text search
• 📄 Paginated results
• 🏷️ Topic tags

Opportunities Manager

• 🚨 Urgency-based filtering
• ✉️ One-click email generation
• 📅 Meeting calendar integration

Settings Panel

• ⚙️ Configure target states
• 📝 Select policy topics
• 🔔 Email notifications
• 📊 Agent status monitoring

---

API Endpoints

Core Endpoints

GET  /api/health                    - Health check
GET  /api/dashboard                 - Dashboard statistics
GET  /api/opportunities             - List opportunities (filterable)
GET  /api/documents                 - List documents (searchable)
POST /api/workflow/start            - Start analysis workflow
GET  /api/workflow/{id}/status      - Check workflow status
POST /api/advocacy/email/{id}       - Generate advocacy email
GET  /api/settings                  - Get settings
PUT  /api/settings                  - Update settings
GET  /api/agents/status             - Agent health status

API Documentation

Once deployed, access interactive API docs at:

• Swagger UI: https://your-app-url/api/docs
• ReDoc: https://your-app-url/api/redoc

---

Monitoring

View App Logs

databricks apps logs open-navigator

Check App Status

databricks apps get open-navigator

Monitor Model Serving

databricks serving-endpoints get policy-classifier-prod
databricks serving-endpoints get sentiment-analyzer-prod

---

Troubleshooting

Issue: "Frontend not built"

Solution:

cd frontend
npm install
npm run build

Issue: "Databricks CLI not found"

Solution:

pip install databricks-cli
databricks configure --token

Issue: "Secrets not accessible"

Solution:

# Recreate secrets scope
databricks secrets create-scope --scope oral-health-app
databricks secrets put --scope oral-health-app --key host --string-value "$DATABRICKS_HOST"
databricks secrets put --scope oral-health-app --key openai_key --string-value "$OPENAI_API_KEY"

Issue: "App deployment failed"

Check logs:

databricks apps logs open-navigator --follow

---

Cost Optimization

Databricks Apps Pricing

Component	Cost Estimate
App hosting	$0.10-0.30/hour
Model Serving (Small)	$0.10-0.50/hour
Delta Lake storage	$0.023/GB/month
SQL Warehouse	Pay per query

Total estimated cost: ~$50-150/month for moderate usage

Cost Savings Tips

1. Scale-to-zero: Model serving endpoints automatically scale down when idle
2. Batch processing: Process documents in batches rather than real-time
3. Hybrid classification: Use keyword matching before LLM calls (saves ~70% LLM costs)
4. Delta Lake optimization: Enable auto-compaction and Z-ordering

---

Security

Authentication

Databricks Apps automatically integrate with:

• ✅ Workspace SSO
• ✅ OAuth 2.0
• ✅ SCIM user provisioning

Data Access

All data access is governed by:

• ✅ Unity Catalog permissions
• ✅ Row-level security
• ✅ Column-level masking

Secrets Management

Sensitive credentials stored in:

• ✅ Databricks Secrets
• ✅ Never in code or logs
• ✅ Automatic rotation support

---

Next Steps

1. Deploy Model Serving Endpoints

   python -m databricks.deployment

2. Initialize Delta Lake Tables

   -- Run in Databricks SQL
   CREATE SCHEMA IF NOT EXISTS main.agents;

3. Start Data Ingestion

   • Configure target municipalities
   • Run initial scraping workflow
   • Monitor agent status

4. Customize UI

   • Edit frontend components in frontend/src/
   • Rebuild: npm run build
   • Redeploy: ./scripts/deploy-databricks-app.sh

---

Support

• 📖 Documentation: See README.md and DATABRICKS_MIGRATION.md
• 🐛 Issues: Report via GitHub Issues
• 💬 Community: Join discussions

---

License

MIT License - See LICENSE file for details