• 2025-08 — Added a demo of II-Search-CIR-4B, our 4B search-reasoning model with Code-Integrated Reasoning. See examples/ii_search_4b.
• 2025-06 — II-Researcher scores 84.12% on Google's Frames benchmark using DeepSeek-R1-0528. Details below ↓

• Overview
• Features
• Benchmarks
• Demo
• Architecture
• Quick Start
• Usage
  • CLI
  • Web Interface
  • MCP (Claude Desktop)
  • Docker
• Advanced Configuration
• Contributing
• License
• Citation
• Acknowledgments

II-Researcher is a deep search agent built by Intelligent Internet. Given a question, it autonomously searches the web, visits and reads pages, reflects on intermediate findings, and synthesizes a comprehensive final answer with references — the same workflow a human researcher would follow, fully automated.

It ships as a complete, self-hostable stack: a Python library and CLI, a streaming FastAPI backend, a Next.js web UI, and an MCP server that plugs directly into Claude and other MCP-compatible clients.

For the design story and evaluation details, read our blog post.

• 🔍 Intelligent web search with pluggable providers (Tavily, SerpAPI, Jina)
• 🕸️ Web scraping and content extraction with multiple providers (Firecrawl, Browser, BeautifulSoup, Tavily Extract, Jina), including PDF and YouTube handling
• 🧠 Multi-step reasoning and reflection — the agent thinks, acts, and self-corrects until it has enough evidence
• 🗜️ Context compression — embedding-based and LLM-based compressors keep long pages within the model's context window
• ⚙️ Model-agnostic via LiteLLM — run OpenAI, DeepSeek, Gemini, OpenRouter, or any OpenAI-compatible endpoint (including self-hosted models)
• ⚡ Fully asynchronous with token-level streaming end to end (CLI, SSE API, and web UI)
• 📝 Referenced report generation with selectable basic and advanced report types and multilingual support
• 🔌 MCP server for one-line integration with Claude Desktop and other MCP clients
• 🧱 Structured outputs powered by BAML

We benchmarked II-Researcher on the Frames dataset by Google using the DeepSeek-R1-0528 model, achieving an accuracy of 84.12%.

Web interface:

MCP integration with Claude:

flowchart TB
    CLI["CLI"]
    WEB["Web UI (Next.js)"]
    MCPS["MCP Server"]
    API["FastAPI Backend (SSE streaming)"]

    WEB --> API
    CLI --> AGENT
    API --> AGENT
    MCPS --> AGENT

    subgraph AGENT["Reasoning Agent"]
        THINK["Multi-step reasoning loop:<br/>think → act → reflect"]
    end

    AGENT --> SEARCH["Web Search<br/>SerpAPI · Tavily · Jina"]
    AGENT --> SCRAPE["Page Visit<br/>Firecrawl · Browser · BeautifulSoup · Tavily Extract · Jina"]
    SCRAPE --> COMPRESS["Context Compression<br/>embedding-based + LLM-based"]
    SEARCH --> AGENT
    COMPRESS --> AGENT

    AGENT --> REPORT["Report Builder<br/>referenced answer · basic / advanced"]
    LITELLM["LiteLLM Proxy<br/>any OpenAI-compatible model"] -.-> AGENT

How it works:

1. The reasoning agent receives a question and enters a think–act–reflect loop, streaming its reasoning tokens as it goes.
2. At each step it can call tools: web search (SerpAPI, Tavily, or Jina) and page visit (Firecrawl, headless browser, BeautifulSoup, Tavily Extract, or Jina — with dedicated handling for PDFs and YouTube).
3. Retrieved content passes through context compression (embedding similarity filtering, optionally an LLM compressor) so only the most relevant material enters the model's context.
4. When the agent has gathered enough evidence, the report builder writes the final answer with references, in basic or advanced form.
5. All model calls route through a LiteLLM proxy, so any OpenAI-compatible model — commercial or self-hosted — can power every stage.

• Python 3.10+ (for local development)
• Docker and Docker Compose (for containerized deployment)
• Node.js and npm (for local frontend development)

From PyPI:

pip install ii-researcher

Or from source:

git clone https://github.com/Intelligent-Internet/ii-researcher.git
cd ii-researcher
pip install -e .

# API Keys
export OPENAI_API_KEY="your-openai-api-key"
export TAVILY_API_KEY="your-tavily-api-key"       # required when SEARCH_PROVIDER=tavily
export SERPAPI_API_KEY="your-serpapi-api-key"     # required when SEARCH_PROVIDER=serpapi
export FIRECRAWL_API_KEY="your-firecrawl-api-key" # required when SCRAPER_PROVIDER=firecrawl

# API Endpoints
export OPENAI_BASE_URL="http://localhost:4000"

# Search and scraping providers
export SEARCH_PROVIDER="serpapi"    # Options: 'serpapi' | 'tavily' | 'jina'
export SCRAPER_PROVIDER="firecrawl" # Options: 'firecrawl' | 'bs' | 'browser' | 'tavily_extract' | 'jina'

# Context compression
export COMPRESS_EMBEDDING_MODEL="text-embedding-3-large"
export COMPRESS_SIMILARITY_THRESHOLD="0.3"
export COMPRESS_MAX_OUTPUT_WORDS="4096"
export COMPRESS_MAX_INPUT_WORDS="32000"

# Timeouts and performance
export SEARCH_PROCESS_TIMEOUT="300" # seconds
export SEARCH_QUERY_TIMEOUT="20"    # seconds
export SCRAPE_URL_TIMEOUT="30"      # seconds
export STEP_SLEEP="100"             # milliseconds

# Reasoning models
export R_MODEL=r1                # model used for reasoning
export R_TEMPERATURE=0.2         # temperature for the reasoning model
export R_REPORT_MODEL=gpt-4o     # model used for writing the final report
export R_PRESENCE_PENALTY=0      # presence_penalty for the reasoning model

Optional — LLM-based context compression (better compression quality):

export USE_LLM_COMPRESSOR="TRUE"
export FAST_LLM="gemini-lite" # model used for context compression

pip install litellm

Create litellm_config.yaml:

model_list:
  - model_name: text-embedding-3-large
    litellm_params:
      model: text-embedding-3-large
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: r1
    litellm_params:
      model: deepseek-reasoner
      api_base: https://api.deepseek.com/beta
      api_key: os.environ/DEEPSEEK_API_KEY

litellm_settings:
  drop_params: true

Then start the server (defaults to http://localhost:4000):

litellm --config litellm_config.yaml

model_list:
  - model_name: text-embedding-3-large
    litellm_params:
      model: text-embedding-3-large
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: openai/chatgpt-4o-latest
      api_base: https://openrouter.ai/api/v1
      api_key: your_openrouter_api_key_here
  - model_name: r1
    litellm_params:
      model: deepseek/deepseek-r1
      api_base: https://openrouter.ai/api/v1
      api_key: your_openrouter_api_key_here
  - model_name: gemini-lite
    litellm_params:
      model: gemini/gemini-2.5-pro-preview-03-25
      api_base: https://openrouter.ai/api/v1
      api_key: your_openrouter_api_key_here

litellm_settings:
  drop_params: true

python ii_researcher/cli.py --question "your question here" --stream

python ii_researcher/cli.py --question "your question here" --stream

Useful flags:

1. Start the backend API:

python api.py

The API server runs on http://localhost:8000 and exposes an SSE endpoint (/search) that streams every step of the research process.

2. Configure the frontend — create frontend/.env:

NEXT_PUBLIC_API_URL=http://localhost:8000

3. Install and run the frontend:

cd frontend
npm install
npm run dev

The web UI is now available at http://localhost:3000.

Connect II-Researcher to Claude Desktop (or any MCP client) as a research tool:

1. Set up your environment file:

cp .env.example .env
# Edit .env and add your API keys

2. Install the MCP server into Claude:

mcp install mcp/server.py -f .env

3. Restart your Claude App. See Claude Desktop Integration for a walkthrough of the integration flow.

Make sure your environment variables are set (step 2 of Quick Start), then:

docker compose up --build -d

This starts three services:

# View logs
docker compose logs -f            # all services
docker compose logs -f api        # a single service

# Stop everything
docker compose down

python3 -m sglang.launch_server --model-path Qwen/QwQ-32B --host 0.0.0.0 --port 30000 --tp 8 --context-length 131072

export STRATEGIC_LLM="gpt-4o" # model used to choose the next action
export SMART_LLM="gpt-4o"     # model used for other pipeline tasks

We welcome contributions of all kinds — bug reports, feature requests, documentation, and code. Please read our Contributing Guide to get started, and see our Code of Conduct and Security Policy.

II-Researcher is released under the Apache License 2.0.

If you use II-Researcher in your research, please cite:

@misc{ii-researcher2025,
  title        = {II-Researcher: An Open-Source Deep Search Agent},
  author       = {Intelligent Internet},
  year         = {2025},
  howpublished = {\url{https://github.com/Intelligent-Internet/ii-researcher}}
}

II-Researcher is inspired by and built with the support of the open-source community:

• LiteLLM — efficient AI model integration
• node-DeepResearch — prompt inspiration
• gpt-researcher — prompt inspiration, web scraper tool
• BAML — structured outputs