Upgrade Firecrawl loader to firecrawl-py v2 (v4 SDK) + add search()#4095
Upgrade Firecrawl loader to firecrawl-py v2 (v4 SDK) + add search()#4095rakshith48 wants to merge 2 commits into
Conversation
Migrate camel/loaders/firecrawl_reader.py from the legacy v1 firecrawl-py
SDK to the current v2 SDK (firecrawl-py v4). Public method names and
signatures on the `Firecrawl` wrapper are unchanged so existing callers
keep working; only the underlying SDK calls change.
- Import `from firecrawl import Firecrawl as FirecrawlApp` (v2 unified
client) instead of the v1 `FirecrawlApp`.
- crawl_url -> crawl, check_crawl_status -> get_crawl_status,
scrape_url -> scrape, map_url -> map.
- structured_scrape now uses the v2 `json` scrape format
(formats=[{"type": "json", "schema": ...}]) and reads `json` instead of
the v1 `extract` field.
- v2 returns typed pydantic models (Document, CrawlJob, MapData); normalize
them back to dict/list so the public return shapes are preserved
(map_site still returns a list of URL strings).
- Bump the firecrawl-py pin to `>=4.0.0,<5` in pyproject.toml.
- Update test/loaders/test_firecrawl.py to the v2 API (patch
`firecrawl.Firecrawl`, mock crawl/get_crawl_status/scrape/map) and add
structured_scrape success coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@rakshith48 thanks，could you please also open a related issue？ |
|
Done, opened #4100 and linked it to this PR ( |
3 participants
Resolves #4100
Upgrades
camel/loaders/firecrawl_reader.pyfrom firecrawl-py v1 to v2 (v4 SDK):FirecrawlApp→Firecrawl, v2 methods + typed-response handling, pinfirecrawl-py>=4,<5. Public wrapper method names/signatures preserved (scrape/crawl/structured_scrape/map_site); adds a newsearch()method. Unit tests migrated to the v2 API.Verification: static + SDK-introspection/mocked against the real v2 SDK (
firecrawl-py 4.28.2/@mendable firecrawl v4); not run against the live API. Happy to address review/CI feedback.🤖 Generated with Claude Code