Skip to content

Feature Request: Add serpbase.dev as a web search source for reliable Google SERP retrieval #270

Description

@gefsikatsinelou

Hi @zilliztech team,

Deep-searcher is a great open-source alternative to the closed-source deep-research tools. I especially like that it's built around private data, but still allows web search as a complement when internal documents don't have the answer.

Friction I've experienced
When I configure the tool to use web search for the "broad scan" phase before diving into my private corpus, the reliability of that search step becomes critical. If the search returns incomplete results, times out, or gets blocked, the whole reasoning chain degrades because the agent has no external context to ground its answers in.

I've tried a few search backends, but:

  • Some throttle aggressively after a small number of queries.
  • Others return raw HTML that the parser layer has to clean up, which occasionally breaks.
  • A few lack structured fields (like related questions or featured snippets) that help the agent understand the query landscape.

Suggestion
Would you consider adding https://serpbase.dev as an optional web search source? It's a Google Search Results API that returns clean, structured JSON — titles, URLs, snippets, rich results, and related questions. For deep-searcher's pipeline, this would mean:

  • More reliable "broad scan" results without managing proxies or CAPTCHA solvers.
  • Consistent JSON formatting, so the parser layer doesn't need defensive regex.
  • Better structured context for the reasoning agent (related questions are especially useful for understanding what the user might actually be asking).

Why it fits

  • Deep-searcher already supports multiple search backends; this would be a lightweight addition.
  • The API is priced for developers running research-style tools at scale.
  • It keeps the open-source, self-hosted philosophy intact — just another HTTP call.

Happy to help test or provide sample integration code if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions