Skip to main content

Unified Nonprofit Data Management

Single Source of Truth: data/gold/nonprofits_organizations.parquet

🎯 New Workflow​

✅ DO THIS (Single Unified File)​

# Check stats
python scripts/manage_nonprofits.py stats

# Enrich a subset (updates main file in place)
python scripts/manage_nonprofits.py enrich-990 --states AL --sample 100

# Enrich specific orgs
python scripts/manage_nonprofits.py enrich-990 --ein-list eins.txt

# Enrich all Alabama + Michigan health nonprofits
python scripts/manage_nonprofits.py enrich-990 --states AL MI --ntee E

# BigQuery enrichment
python scripts/manage_nonprofits.py enrich-bigquery --states AL
# ... run SQL in BigQuery web UI, export CSV ...
python scripts/manage_nonprofits.py merge-bigquery

❌ DON'T DO THIS (Creates Separate Files)​

# OLD WAY - Creates proliferation of files
python scripts/enrich_nonprofits_gt990.py \
    --input data/gold/nonprofits_tuscaloosa.parquet \
    --output data/gold/nonprofits_tuscaloosa_form990.parquet  # ❌ Extra file!

📊 Key Commands​

Show Statistics​

python scripts/manage_nonprofits.py stats

Output:

📊 TOTAL: 1,952,238 organizations
💰 ENRICHMENT STATUS:
   Form 990 data: 307 (0.0%)
   BigQuery data: 0 (0.0%)
📝 MISSION STATEMENTS:
   At least one source: 299 (0.0%)

Enrich Incrementally​

By State:

python scripts/manage_nonprofits.py enrich-990 --states AL
# Updates main file with 62K Alabama nonprofits enriched

By NTEE Category:

python scripts/manage_nonprofits.py enrich-990 --ntee E
# Updates main file with 45K health nonprofits enriched

Combined Filters:

python scripts/manage_nonprofits.py enrich-990 --states AL MI --ntee E
# Updates main file with Alabama + Michigan health orgs

Test on Sample:

python scripts/manage_nonprofits.py enrich-990 --sample 100
# Updates main file with 100 random orgs enriched (for testing)

Specific EINs:

# Create file with EINs (one per line)
echo "631024890" > eins.txt
echo "631041304" >> eins.txt

python scripts/manage_nonprofits.py enrich-990 --ein-list eins.txt

🔄 How In-Place Updates Work​

  1. Load full dataset (1.9M orgs)
  2. Filter to subset (e.g., Alabama = 62K orgs)
  3. Enrich only the filtered subset
  4. Merge back:
    • Remove old data for those EINs
    • Add newly enriched data
    • Sort by EIN
  5. Save back to same file

Result: Only one file, incrementally enriched!

🗑️ Cleanup Old Files​

Preview Cleanup (Dry Run)​

python scripts/cleanup_nonprofit_files.py

Output:

🔄 Found enriched data: nonprofits_tuscaloosa_form990.parquet
   ✅ Merged 921 enriched organizations

🗑️  Found 9 file(s) to clean up:
   - nonprofits_tuscaloosa.parquet (0.0 MB)
   - nonprofits_tuscaloosa_form990.parquet (0.1 MB)
   - /tmp/test_*.parquet (0.1 MB)
   
⚠️  DRY RUN - No files will be deleted
   Run with --execute to actually delete files

Execute Cleanup​

python scripts/cleanup_nonprofit_files.py --execute

What it does:

  • ✅ Merges any enrichment from old files into main file
  • ✅ Deletes old/redundant files
  • ✅ Leaves only data/gold/nonprofits_organizations.parquet

📁 File Organization​

✅ Keep (Single Source of Truth)​

data/gold/nonprofits_organizations.parquet  # 1.9M orgs, incrementally enriched

🗑️ Remove (Old Workflow)​

data/gold/nonprofits_tuscaloosa.parquet              # ❌ Subset
data/gold/nonprofits_tuscaloosa_form990.parquet      # ❌ Enriched subset  
data/gold/nonprofits_990_enriched.parquet            # ❌ Another version
/tmp/test_*.parquet                                   # ❌ Test files

🎨 Progressive Enrichment Strategy​

Enrich the dataset progressively to avoid overwhelming API limits:

Phase 1: Test Sample (TODAY)​

python scripts/manage_nonprofits.py enrich-990 --sample 1000
# Test with 1K random orgs

Phase 2: Priority States (WEEK 1)​

python scripts/manage_nonprofits.py enrich-990 --states AL MI
# Enrich Alabama + Michigan (118K orgs)

Phase 3: Priority NTEE (WEEK 2)​

python scripts/manage_nonprofits.py enrich-990 --ntee E P
# Health + Human Services (199K orgs)

Phase 4: Remaining States (MONTH 1)​

# Enrich 5-10 states per day
for state in CA TX NY FL PA; do
    python scripts/manage_nonprofits.py enrich-990 --states $state
    echo "Completed $state"
    sleep 3600  # Wait 1 hour between states
done

Phase 5: BigQuery Layer (MONTH 2)​

# Add missions + websites from BigQuery
python scripts/manage_nonprofits.py enrich-bigquery
# ... export CSV ...
python scripts/manage_nonprofits.py merge-bigquery

🔍 Advanced Usage​

Check Enrichment Status by State​

python -c "
import pandas as pd
df = pd.read_parquet('data/gold/nonprofits_organizations.parquet')
enriched = df[df['form_990_status'] == 'found']
by_state = enriched.groupby('state').size().sort_values(ascending=False).head(10)
print('Top 10 states by Form 990 coverage:')
print(by_state)
"

Export EINs Needing Enrichment​

python -c "
import pandas as pd
df = pd.read_parquet('data/gold/nonprofits_organizations.parquet')
# Alabama health orgs without 990 data
needs_enrichment = df[
    (df['state'] == 'AL') & 
    (df['ntee_code'].str.startswith('E', na=False)) &
    (df['form_990_status'].isna())
]['ein']
needs_enrichment.to_csv('alabama_health_needs_enrichment.txt', index=False, header=False)
print(f'Exported {len(needs_enrichment)} EINs')
"

💡 Benefits of Unified File​

  1. ✅ Single source of truth - No confusion about which file is current
  2. ✅ Incremental updates - Add enrichment data without duplicating base data
  3. ✅ Smaller disk usage - No duplicate base data across files
  4. ✅ Easier tracking - One file to track, version, backup
  5. ✅ Simpler workflows - No need to manage file versions
  6. ✅ Better for git - One file to track changes in (with git-lfs)

📝 Migration Checklist​

  • Merge existing enrichment data into main file
  • Verify enrichment was merged (run stats)
  • Clean up old files with --execute
  • Update any scripts/docs referencing old files
  • Test new workflow with small sample
  • Document team workflow

🆘 Troubleshooting​

"Main file not found"​

# Create main file from EO-BMF data
python pipeline/create_gold_tables.py --nonprofits-only

"Lost my enrichment data!"​

Don't worry! Run cleanup script first (without --execute) - it will merge any enrichment data before deleting files.

"Want to keep a backup"​

# Backup before cleanup
cp data/gold/nonprofits_organizations.parquet \
   data/gold/nonprofits_organizations.$(date +%Y%m%d).parquet.bak

📚 Related Documentation​