Skip to main content

Automated eBoard Scraping Solutions

This guide covers fully automated solutions to bypass Incapsula protection without manual cookie extraction.


Summary of Options​

SolutionCostDifficultySuccess RateSpeed
1. Undetected ChromeDriverFreeEasy70-85%Medium
2. Playwright + Residential Proxies$10-50/monthMedium90-95%Fast
3. Browser Automation Services$30-100/monthEasy95-99%Fast
4. Captcha Solving Service$1-3/1000 solvesMedium85-90%Slow

Why It Works​

undetected-chromedriver patches Selenium to bypass bot detection:

  • Removes navigator.webdriver flag
  • Uses real Chrome binary (not ChromeDriver)
  • Randomizes browser fingerprints
  • Avoids common detection patterns

Installation​

source .venv/bin/activate
pip install undetected-chromedriver

Usage​

# Run the new scraper
python agents/scraper_undetected.py

Or integrate into main scraper:

python main.py scrape \
--state AL \
--municipality "Tuscaloosa City Schools" \
--url http://simbli.eboardsolutions.com/index.aspx?s=2088 \
--platform eboard \
--use-undetected \
--max-events 0

Pros​

  • ✅ Free
  • ✅ No external services required
  • ✅ Works for most Incapsula sites
  • ✅ Easy to implement

Cons​

  • ❌ May still fail on very strict Incapsula settings
  • ❌ Requires GUI environment (can't run headless on some systems)
  • ❌ Slower than Playwright

Option 2: Residential Proxies (Best Success Rate)​

Why It Works​

Incapsula detects datacenter IPs. Residential proxies route through real home IPs that appear legitimate.

BrightData (formerly Luminati)

SmartProxy

Oxylabs

Implementation​

# Install
pip install playwright

# Configure proxy in scraper
async with async_playwright() as p:
browser = await p.chromium.launch(
proxy={
'server': 'http://proxy.smartproxy.com:10000',
'username': 'your_username',
'password': 'your_password'
}
)
# ... rest of scraping code

Add to agents/scraper.py​

# In _scrape_eboard method, add:
import os

proxy_config = None
if os.getenv('RESIDENTIAL_PROXY_URL'):
proxy_config = {
'server': os.getenv('RESIDENTIAL_PROXY_URL'),
'username': os.getenv('PROXY_USERNAME'),
'password': os.getenv('PROXY_PASSWORD')
}

browser = await p.chromium.launch(
proxy=proxy_config,
headless=True
)

.env Configuration​

# Add to .env file
RESIDENTIAL_PROXY_URL=http://proxy.smartproxy.com:10000
PROXY_USERNAME=your_username
PROXY_PASSWORD=your_password

Pros​

  • ✅ Highest success rate (95%+)
  • ✅ Works on any Incapsula configuration
  • ✅ Can run headless
  • ✅ Fast and reliable

Cons​

  • ❌ Costs money ($10-50/month for small projects)
  • ❌ Requires account setup
  • ❌ May have usage limits

Option 3: Browser Automation Services (Easiest)​

Why It Works​

These services run real browsers in the cloud and handle all anti-bot evasion automatically.

Browserless.io

from playwright.async_api import async_playwright

async with async_playwright() as p:
browser = await p.chromium.connect(
'wss://chrome.browserless.io?token=YOUR_TOKEN'
)
page = await browser.new_page()
await page.goto('https://simbli.eboardsolutions.com/...')

ScrapingBee

import requests

response = requests.get(
'https://app.scrapingbee.com/api/v1/',
params={
'api_key': 'YOUR_API_KEY',
'url': 'https://simbli.eboardsolutions.com/...',
'render_js': 'true',
'premium_proxy': 'true'
}
)
content = response.text

Apify

  • Cost: $49/month
  • Pre-built scrapers for common sites
  • Can create custom scrapers
  • https://apify.com

Pros​

  • ✅ Fully managed (no maintenance)
  • ✅ Very high success rate
  • ✅ Handles updates to anti-bot automatically
  • ✅ Can scale easily

Cons​

  • ❌ Most expensive option
  • ❌ Requires external service dependency
  • ❌ May have rate limits

Option 4: Captcha Solving Service​

Why It Works​

If Incapsula shows a CAPTCHA, these services solve it automatically using AI or human workers.

2Captcha

Anti-Captcha

Implementation​

pip install 2captcha-python
from twocaptcha import TwoCaptcha
import os

solver = TwoCaptcha(os.getenv('2CAPTCHA_API_KEY'))

# When Incapsula shows CAPTCHA
try:
result = solver.recaptcha(
sitekey='SITE_KEY_FROM_PAGE',
url='https://simbli.eboardsolutions.com/...'
)

# Inject solution into page
await page.evaluate(f'document.getElementById("g-recaptcha-response").innerHTML="{result["code"]}";')
await page.click('button[type="submit"]')

except Exception as e:
logger.error(f"CAPTCHA solving failed: {e}")

Pros​

  • ✅ Solves CAPTCHAs automatically
  • ✅ Relatively cheap
  • ✅ Works with existing scraper

Cons​

  • ❌ Only useful if CAPTCHA appears
  • ❌ Slower (10-30 seconds per solve)
  • ❌ Not 100% success rate
  • ❌ Costs money per use

Option 5: Reverse Engineer the API​

Why It Works​

eBoard likely has backend APIs that mobile apps or internal tools use. These APIs may have weaker protection.

How to Find APIs​

  1. Use browser DevTools:

    # Open eBoard site in Chrome
    # Press F12 → Network tab
    # Look for XHR/Fetch requests
    # Check requests to:
    # - /api/
    # - .ashx files
    # - .asmx files (SOAP endpoints)
  2. Check for mobile app:

    • Search App Store / Google Play for "eBoard Solutions"
    • Decompile APK to find API endpoints
    • Use mitmproxy to intercept app traffic
  3. Look for GraphQL/REST endpoints:

    curl -I https://simbli.eboardsolutions.com/api/meetings
    curl -I https://simbli.eboardsolutions.com/graphql

Example (if API exists)​

import httpx

# Hypothetical API endpoint
async with httpx.AsyncClient() as client:
response = await client.get(
'https://simbli.eboardsolutions.com/api/v1/meetings',
params={'school_id': 2088},
headers={'User-Agent': 'eBoard-Mobile/1.0'}
)
meetings = response.json()

Pros​

  • ✅ Fastest option
  • ✅ No bot detection
  • ✅ Free
  • ✅ Most reliable

Cons​

  • ❌ Requires reverse engineering skills
  • ❌ API may not exist
  • ❌ API may require authentication
  • ❌ May violate Terms of Service

For Personal/Research Projects (Free)​

Start with Option 1 (Undetected ChromeDriver)

# Install
pip install undetected-chromedriver

# Run test
python agents/scraper_undetected.py

If that fails, use manual cookies (current approach) as fallback.

For Production/Reliable Scraping ($)​

Use Option 2 (Residential Proxies)

Budget: ~$15-75/month depending on volume

Best provider for this use case: SmartProxy ($75/month for 5GB)

# Sign up at smartproxy.com
# Add credentials to .env
# Enable proxy in scraper

RESIDENTIAL_PROXY_URL=http://proxy.smartproxy.com:10000
PROXY_USERNAME=your_username
PROXY_PASSWORD=your_password

For Large Scale / Enterprise​

Use Option 3 (Browserless.io or ScrapingBee)

Budget: $40-100/month

Most reliable, fully managed solution.


Implementation Plan​

Phase 1: Try Free Options​

  1. ✅ Install undetected-chromedriver
  2. ✅ Test on Tuscaloosa City Schools
  3. ✅ Measure success rate over 10 runs
  4. If success rate > 80%, use this going forward

Phase 2: Add Proxy Support (If Phase 1 Fails)​

  1. Add proxy configuration to existing Playwright scraper
  2. Sign up for SmartProxy trial
  3. Test with residential proxy
  4. If successful, add to production

Phase 3: Optimize​

  1. Add retry logic with exponential backoff
  2. Rotate between different methods
  3. Cache successful cookies for reuse
  4. Monitor success rate and adjust

Next Steps​

Would you like me to:

  1. Integrate undetected-chromedriver into the main scraper (1-click solution)
  2. Add residential proxy support to existing code (requires proxy account)
  3. Try to reverse engineer the eBoard API (advanced, may take time)
  4. Create a hybrid approach that tries multiple methods automatically

Let me know which direction you'd prefer!