Skip to main content

React + FastAPI Databricks App Refactoring

Executive Summary​

The Oral Health Policy Pulse has been completely refactored from a CLI-based multi-agent system into a modern full-stack web application that deploys as a Databricks App.

What Changed​

Before (v1.0)After (v2.0)
❌ CLI-only interface✅ Modern React web UI
❌ Manual command execution✅ Interactive dashboard
❌ Text-based outputs✅ Visual charts & maps
❌ Local deployment only✅ Cloud-native Databricks Apps
❌ Static HTML heatmap✅ Interactive Leaflet map
❌ No user settings✅ Configurable UI settings

New Architecture​

Frontend (React + TypeScript)​

Technology Stack:

  • React 18.2 with TypeScript
  • Vite for build tooling
  • Tailwind CSS for styling
  • React Router for navigation
  • TanStack Query for data fetching
  • Recharts for data visualization
  • Leaflet for interactive maps

Pages Created:

  1. Dashboard (frontend/src/pages/Dashboard.tsx)

    • Real-time statistics cards
    • Topic distribution bar/pie charts
    • Recent opportunities table

  2. Interactive Heatmap (frontend/src/pages/Heatmap.tsx)

    • Geographic visualization using Leaflet
    • State and topic filters
    • Color-coded urgency markers
    • Popup details for each opportunity

  3. Documents Browser (frontend/src/pages/Documents.tsx)

    • Full-text search
    • Pagination
    • Topic tags
    • Direct links to source documents

  4. Opportunities Manager (frontend/src/pages/Opportunities.tsx)

    • Filterable list view
    • Urgency-based sorting
    • One-click email generation
    • Talking points display
    • Confidence score visualization

  5. Settings Panel (frontend/src/pages/Settings.tsx)

    • Target state configuration
    • Policy topic selection
    • Confidence threshold slider
    • Email notification preferences
    • Agent status monitoring

Components:

  • Layout (frontend/src/components/Layout.tsx)
    • Sidebar navigation
    • Header with breadcrumbs
    • Responsive design

Backend (FastAPI Refactored)​

New File: api/app.py

Completely rewritten FastAPI application optimized for:

  • ✅ Serving React static files (SPA routing)
  • ✅ REST API endpoints for all frontend interactions
  • ✅ Databricks Apps compatibility
  • ✅ Delta Lake integration
  • ✅ Background task processing
  • ✅ CORS support for local development

Key Endpoints:

GET  /api/health                    # Health check
GET  /api/dashboard                 # Dashboard stats
GET  /api/opportunities             # List opportunities
GET  /api/documents                 # Browse documents
POST /api/workflow/start            # Start analysis
POST /api/advocacy/email/{id}       # Generate email
GET  /api/settings                  # Get settings
PUT  /api/settings                  # Update settings
GET  /api/agents/status             # Agent status

Static File Serving:

  • React build output served from api/static/
  • SPA routing with catchall route
  • Automatic index.html fallback

Database Layer​

New File: pipeline/delta_lake_queries.py

Added async query methods to support web app:

async def get_dashboard_stats()       # Dashboard statistics
async def query_opportunities()       # Filtered opportunity queries
async def query_documents()           # Document search/pagination
async def count_documents()           # Total count for pagination
async def get_opportunity()           # Single opportunity details

Databricks Apps Integration​

Configuration Files​

1. app.yaml (Databricks Apps manifest)

command:
  - "uvicorn"
  - "api.app:app"
  - "--host"
  - "0.0.0.0"
  - "--port"
  - "8000"

env:
  - name: DATABRICKS_HOST
    valueFrom:
      databricksSecret:
        key: host
        scope: oral-health-app
  - name: OPENAI_API_KEY
    valueFrom:
      databricksSecret:
        key: openai_key
        scope: oral-health-app

resources:
  - name: policy-classifier-endpoint
    modelServing:
      endpoint: policy-classifier-prod

port: 8000

2. Dockerfile.app (Container image for Databricks Apps)

  • Multi-stage build
  • Node.js for frontend build
  • Python 3.11 for backend
  • Optimized for CPU-only deployment

Deployment Scripts​

1. scripts/deploy-databricks-app.sh

# Automated deployment script:
# 1. Build React frontend
# 2. Create Databricks secrets
# 3. Deploy to Databricks Apps
# 4. Provide access URL

2. scripts/setup-local.sh

# Local development setup:
# - Install Python deps
# - Install npm packages
# - Create .env file

3. scripts/test-app.py

# Test production build locally
# Simulates Databricks Apps environment

Project Structure​

oral-health-policy-pulse/
├── frontend/                      # NEW: React application
│   ├── src/
│   │   ├── components/
│   │   │   └── Layout.tsx        # Main layout component
│   │   ├── pages/
│   │   │   ├── Dashboard.tsx     # Dashboard page
│   │   │   ├── Heatmap.tsx       # Interactive map
│   │   │   ├── Documents.tsx     # Document browser
│   │   │   ├── Opportunities.tsx # Opportunity manager
│   │   │   └── Settings.tsx      # Settings panel
│   │   ├── App.tsx               # Root component
│   │   ├── main.tsx              # Entry point
│   │   └── index.css             # Tailwind styles
│   ├── package.json              # npm dependencies
│   ├── vite.config.ts            # Vite configuration
│   ├── tsconfig.json             # TypeScript config
│   ├── tailwind.config.js        # Tailwind config
│   └── index.html                # HTML template
│
├── api/
│   ├── app.py                    # NEW: Refactored FastAPI app
│   ├── main.py                   # LEGACY: Original API
│   └── static/                   # Built React files (generated)
│
├── pipeline/
│   ├── delta_lake.py             # Original Delta Lake pipeline
│   └── delta_lake_queries.py     # NEW: Query methods for web app
│
├── scripts/                       # NEW: Deployment scripts
│   ├── deploy-databricks-app.sh  # Deploy to Databricks
│   ├── setup-local.sh            # Local development setup
│   └── test-app.py               # Test production build
│
├── app.yaml                       # NEW: Databricks Apps config
├── Dockerfile.app                 # NEW: Production Docker image
├── DATABRICKS_APP_GUIDE.md        # NEW: Deployment documentation
└── README.md                      # UPDATED: Added React info

Features by Page​

1. Dashboard​

Statistics Cards:

  • Total documents analyzed
  • Advocacy opportunities found
  • States monitored

Charts:

  • Topic distribution (bar chart)
  • Topic distribution (pie chart)

Recent Activity:

  • Latest opportunities table
  • Sortable and filterable

2. Interactive Heatmap​

Map Features:

  • OpenStreetMap tiles
  • Circle markers for opportunities
  • Color-coded by urgency:
    • 🔴 Critical
    • 🟠 High
    • 🟡 Medium
    • 🟢 Low

Interactivity:

  • Click markers for details
  • Filter by state dropdown
  • Filter by topic dropdown
  • Real-time updates

3. Document Browser​

Features:

  • Full-text search across all documents
  • Pagination (20 per page)
  • Topic tags for each document
  • Direct links to source
  • Meeting date display
  • State/municipality info

4. Opportunities Manager​

Display:

  • Card-based layout
  • Urgency badge
  • Key talking points
  • Confidence score bar

Actions:

  • Generate advocacy email
  • Contact via email
  • Add to calendar
  • Copy talking points

Filters:

  • Critical only
  • High priority
  • Medium priority
  • Low priority

5. Settings Panel​

Configuration Options:

  • Target states (multi-select)
  • Policy topics (checkboxes)
  • Confidence threshold (slider)
  • Email notifications (toggle)

System Status:

  • Agent health indicators
  • Uptime display
  • Real-time status

Development Workflow​

Local Development​

1. Backend (Hot Reload)

source venv/bin/activate
uvicorn api.app:app --reload
# Runs on http://localhost:8000

2. Frontend (Hot Reload)

cd frontend
npm run dev
# Runs on http://localhost:3000
# Proxies API calls to :8000

Production Build​

1. Build Frontend

cd frontend
npm run build
# Outputs to api/static/

2. Test Locally

python scripts/test-app.py
# Serves at http://localhost:8000

3. Deploy to Databricks

./scripts/deploy-databricks-app.sh
# One-command deployment

Benefits of Refactoring​

User Experience​

BeforeAfter
Command-line onlyBeautiful web interface
Manual file managementAutomatic workflows
Static visualizationsInteractive maps & charts
No configuration UIFull settings panel
Email via CLIOne-click generation

Developer Experience​

BeforeAfter
Python-onlyFull-stack (Python + TypeScript)
Local deploymentCloud-native (Databricks Apps)
Manual setupOne-command deployment
Limited monitoringBuilt-in observability
No UI testingComponent testing with Vitest

Operations​

BeforeAfter
Single-user CLIMulti-user web app
Local storageCloud Delta Lake
Manual scalingAuto-scaling
No authenticationDatabricks SSO
Basic loggingEnterprise monitoring

Technology Choices​

Why React?​

  • ✅ Industry-standard UI framework
  • ✅ Rich ecosystem of libraries
  • ✅ TypeScript support for type safety
  • ✅ Excellent developer experience
  • ✅ Easy to maintain and extend

Why Vite?​

  • ✅ Lightning-fast dev server
  • ✅ Instant hot module replacement
  • ✅ Optimized production builds
  • ✅ Native ES modules support
  • ✅ Built-in TypeScript support

Why Tailwind CSS?​

  • ✅ Utility-first approach
  • ✅ Rapid prototyping
  • ✅ Consistent design system
  • ✅ Small production bundle
  • ✅ Highly customizable

Why Databricks Apps?​

  • ✅ Integrated with workspace
  • ✅ Automatic authentication
  • ✅ Unity Catalog integration
  • ✅ Seamless data access
  • ✅ Enterprise-ready

Migration Path​

For Existing Users​

v1.0 (CLI) → v2.0 (Web App)

  1. Keep using CLI - Original functionality preserved in api/main.py
  2. Gradually adopt web UI - Run both simultaneously
  3. Full migration - Switch to web-only when ready

Backward Compatibility:

  • All CLI commands still work
  • Original API endpoints unchanged
  • Can run standalone or web mode

For New Users​

Start with Web UI:

  1. Deploy to Databricks Apps
  2. Access via browser
  3. No CLI needed!

Performance Optimizations​

Frontend​

  • ✅ Code splitting with React lazy loading
  • ✅ Query caching with TanStack Query
  • ✅ Optimized bundle size (~300KB gzipped)
  • ✅ Lazy-loaded charts and maps
  • ✅ Debounced search inputs

Backend​

  • ✅ Async endpoints with FastAPI
  • ✅ Background task processing
  • ✅ Delta Lake query optimization
  • ✅ Static file caching
  • ✅ CORS preflight caching

Deployment​

  • ✅ Scale-to-zero when idle
  • ✅ Auto-scaling under load
  • ✅ CDN for static assets
  • ✅ Connection pooling
  • ✅ Gzip compression

Next Steps​

Planned Features​

Phase 1: Enhanced Analytics

  • Historical trend charts
  • Predictive analytics
  • Custom report builder

Phase 2: Collaboration

  • User comments/notes
  • Shared dashboards
  • Team workspaces

Phase 3: Automation

  • Scheduled reports
  • Email alerts
  • Slack integration

Phase 4: Advanced AI

  • Document Q&A chatbot
  • Meeting video analysis
  • Speech-to-text integration

Deployment Examples​

Example 1: Local Testing​

# Setup
./scripts/setup-local.sh
source venv/bin/activate

# Run backend
uvicorn api.app:app --reload &

# Run frontend
cd frontend && npm run dev

Example 2: Production Deployment​

# Configure
export DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
export DATABRICKS_TOKEN=dapi...
export OPENAI_API_KEY=sk-...

# Deploy
./scripts/deploy-databricks-app.sh

# Monitor
databricks apps logs oral-health-policy-pulse --follow

Example 3: Docker Deployment​

# Build
docker build -f Dockerfile.app -t oral-health-app .

# Run
docker run -p 8000:8000 \
  -e DATABRICKS_HOST=$DATABRICKS_HOST \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  oral-health-app

Conclusion​

The React + FastAPI refactoring transforms Oral Health Policy Pulse from a developer-focused CLI tool into an enterprise-ready web application that can be:

✅ Deployed instantly to Databricks Apps
✅ Used by non-technical users via web browser
✅ Scaled automatically for any load
✅ Monitored comprehensively with built-in observability
✅ Secured enterprise-grade with Databricks SSO

The future of oral health advocacy is here! 🦷✨