Two-engine Python toolkit for scraping any paginated HTML or WordPress AJAX business directory — config-driven, checkpoint-resumable, concurrent fetching, Cloudflare email decode, and professionally formatted three-sheet Excel export. Point either engine at a new directory with a single config file change. No code edits required.

Found this useful? A ⭐ on GitHub helps other developers find it.

• Preview
• What It Does
• Engines
• Use Cases
• Features
• Performance
• What Data You Get
• Quick Start
• Choosing an Engine
• Configuration Reference
• Output
• Runtime Controls
• Auto-Protection Features
• Resuming a Run
• Prerequisites
• Project Structure
• Running Tests
• Troubleshooting
• Known Limitations
• Part of the B2B Lead Toolkit
• License

Terminal — live progress bar	Excel Output — three sheets

See assets/Sample_Output.csv for realistic sample output rows.

1. Reads config.yaml — one file controls the target URL, CSS selectors (HTML engine) or AJAX parameters (WordPress engine), geographic filter, category mapping, rate limits, and output format.
2. Discovers all listings — paginates through every search-results page (HTML engine) or AJAX POST page (WordPress engine) to collect every profile URL or business marker.
3. Fetches profile pages concurrently — up to --workers N threads (default 3) fetch profile pages in parallel, decoding Cloudflare-obfuscated emails automatically.
4. Filters by geography — HTML engine: regex on location text. WordPress engine: lat/lng bounding box from AJAX map markers.
5. Deduplicates — by URL (HTML) or name + postcode composite key (WordPress).
6. Exports to Excel — a styled Data sheet (clean records), a Flagged sheet (excluded records with reason), and a Summary sheet (run metadata).

Everything is config-driven. The Python source files contain zero site-specific strings — every URL, selector, and parameter lives in config.yaml.

Tuning guidance:

• Use --workers 3 (default) for polite scraping on shared-hosting directories.
• Use --workers 5 for faster runs on directories with robust infrastructure.
• Never exceed --workers 8 (enforced by a hard cap in scraper.py).
• Increase delay_min / delay_max in config if you receive HTTP 429 responses.
• profile_timeout_seconds: 6 eliminates stalls on dead company websites — ConnectTimeout returns instantly.

Expected runtime formula:

minutes ≈ total_records / (workers × 12) × delay_avg

See assets/Sample_Output.csv for realistic sample output.

git clone https://github.com/FAAQJAVED/html-directory-scrapers.git
cd html-directory-scrapers

# HTML engine
cd engines/html
pip install -r requirements.txt

# WordPress engine
cd engines/wordpress
pip install -r requirements.txt

cp config.yaml.example config.yaml

Edit config.yaml — set your target URL, CSS selectors (HTML) or AJAX parameters (WordPress), and geographic filter. Every option is annotated in the example file.

cp .env.example .env

Paste session cookies as SCRAPER_COOKIES=... (HTML) or SCRAPER_COOKIES_RAW=... (WordPress). Never stored in config.yaml.

# Standard run
python scraper.py

# Use a different config file
python scraper.py --config my_config.yaml

# Start fresh (clears any saved checkpoint)
python scraper.py --fresh

# Increase concurrent workers for faster scraping
python scraper.py --workers 5

pip install -e ../../    # from either engine folder
html-scraper --config config.yaml
wp-scraper   --config config.yaml

Use the HTML engine if the directory renders its listings directly in the HTML page source — you can see company names and links when you use View Source in your browser.

Use the WordPress engine if you see POST requests to /wp-admin/admin-ajax.php in browser DevTools → Network tab when you trigger a search. The response will be a JSON object containing markers and cards data.

Full option documentation is in config.yaml.example inside each engine folder.

Timestamped console + file logging. Full DEBUG to file, clean INFO to console.

While the scraper is running, press a key — no Enter needed on any platform. Controls are event-driven and respond instantly:

HTML engine only — write to command.txt in the engine folder (useful for headless / cron runs):

echo pause  > command.txt
echo resume > command.txt
echo stop   > command.txt
echo fresh  > command.txt

Both engines save a checkpoint after every listing page. If a run is interrupted for any reason, simply re-run — the checkpoint is detected and the run continues from exactly where it stopped.

python scraper.py           # automatically resumes if checkpoint exists
python scraper.py --fresh   # discard checkpoint and start fresh

No browser driver (Playwright / Selenium) needed. Both engines use pure HTTP requests only.

html-directory-scrapers/
├── assets/                          ← Preview images and sample output
│   ├── Terminal_Preview.png         # Live terminal screenshot
│   ├── Output_Preview.png           # Excel output screenshot
│   └── Sample_Output.csv            # Sample data rows
├── engines/
│   ├── html/                        ← HTML Directory Scraper
│   │   ├── scraper.py               # Thin orchestrator (CLI entry point)
│   │   ├── config.py                # YAML loader + env var injection
│   │   ├── fetcher.py               # httpx client, safe_get, progress, SMTP
│   │   ├── parser.py                # parse_cards, scrape_profile, CF email
│   │   ├── exporter.py              # 3-sheet Excel workbook
│   │   ├── checkpoint.py            # CheckpointManager (atomic JSON)
│   │   ├── controls.py              # command.txt watcher, InputController
│   │   ├── config.yaml.example
│   │   ├── .env.example
│   │   └── requirements.txt
│   └── wordpress/                   ← WordPress Directory Scraper
│       ├── scraper.py               # Thin orchestrator (CLI entry point)
│       ├── config.py                # YAML loader + env var injection
│       ├── fetcher.py               # Session, AJAX POST, nonce, safe_decode
│       ├── parser.py                # parse_cards, filter_by_bounds, profile
│       ├── exporter.py              # 3-sheet Excel workbook
│       ├── checkpoint.py            # CheckpointManager (atomic JSON)
│       ├── controls.py              # InputController (P/R/S/W keyboard)
│       ├── config.yaml.example
│       ├── .env.example
│       └── requirements.txt
├── tests/
│   ├── conftest.py
│   ├── html/
│   │   ├── conftest.py
│   │   └── test_html_engine.py      # 59 tests
│   └── wordpress/
│       ├── conftest.py
│       └── test_wordpress_engine.py # 64 tests
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
├── pyproject.toml
└── README.md

pip install pytest pytest-cov
pytest tests/ -v --cov=engines --cov-report=term-missing

123 tests across both engines. No network calls. No API key required. Full suite runs offline in under 10 seconds.

"Config file not found: config.yaml"

Copy the annotated example and fill in your values:

cp config.yaml.example config.yaml

Scraper runs but saves zero records

1. Open the target URL in your browser and press Ctrl+U (View Source). If you see JavaScript but no company cards, the directory is JS-rendered and not compatible — see Known Limitations.
2. Check DevTools → Network tab for POST requests to /wp-admin/admin-ajax.php. If present, you need the WordPress engine, not the HTML engine.
3. Verify selectors.card_container matches at least one element on the listing page.

Keyboard controls not responding

Press W first — if the stats line prints, the listener is working and the scraper is running normally (a slow site may take 10–30 s per page). If no response, check that your terminal window has focus. On macOS, grant Accessibility permissions: System Settings → Privacy & Security → Accessibility → add your terminal app.

HTTP 429 Too Many Requests

Increase delay_min and delay_max in config.yaml, or reduce --workers to 1:

delay_min: 3.0
delay_max: 6.0

Excel output locked / PermissionError

Close the previous output file in Excel before running — Excel holds an exclusive lock on open .xlsx files.

WordPress engine: "Nonce not found"

The scraper visits register_path to extract the nonce. Check that register_path is the correct path to the directory's search page (not the homepage). Open the page in a browser and search for "nonce" in the HTML source to confirm it is embedded there.

• JavaScript-rendered directories are not supported. If listings only appear after JavaScript executes, neither engine will find any cards. Confirm via View Source before configuring.

• WordPress nonce expiry on very long runs. Nonces typically expire after 24 hours. The engine detects expiry and refreshes automatically. If a run exceeds 24 hours and stops finding records, restart with --fresh.

• Rate limiting is per-worker, not per-run. With --workers 3 and delay_min: 1.0, up to 3 concurrent requests fire per second during profile fetches. Reduce workers if you hit 429 responses.

• SMTP email verification is slow. verify_email: true adds a DNS + SMTP handshake per email. On a 500-record run this adds 10–20 minutes. Use only when email accuracy is critical.

This toolkit is one component of a broader B2B lead generation pipeline.

All tools share the same three-sheet Excel output schema (Data · Flagged · Summary) — results combine directly in Excel or import together into a CRM.

MIT © 2026 FAAQJAVED — see LICENSE