Reproducibility repository for "ScrapeGraphAI-100k: Dataset for Schema-Constrained LLM Generation"

This repository contains the code behind the ScrapeGraphAI-100k paper: dataset analysis scripts, evaluation metrics, the finetuning/distillation pipeline, the PII audit, and the LLM-judge pilot.

The paper releases two datasets and a model:

• scrapegraphai/scrapegraphai-100k — 93,695 real-world schema-constrained extraction events collected via opt-in telemetry of the ScrapeGraphAI library (Q2–Q3 2025), derived from ~9M raw events, deduplicated and balanced for schema diversity (25,453 unique schemas, 8,155 root domains). Each row holds the prompt, Markdown-converted page content, target JSON schema, LLM response, and metadata (model, latency, five schema-complexity metrics, and a structural response_is_valid flag — 96.5% of responses validate, invalid ones are retained for failure-mode research).
• scrapegraphai/scrapegraph-100k-finetuning — a finetuning-ready derivative with (schema, content, response) triples: train (25,244 rows) and test (2,808 rows) with GPT-5-nano regenerated, schema-compliant targets (teacher pseudo-labels), plus a leak-free, human-verified human_eval split (100 rows) for semantic-correctness evaluation.
• scrapegraphai/sgai-qwen3-1.7b — Qwen3-1.7B finetuned on the finetuning dataset.

from datasets import load_dataset

# Full corpus: prompts, content, schemas, responses, complexity metadata
ds = load_dataset("scrapegraphai/scrapegraphai-100k")

# Finetuning derivative: train / test / human_eval splits
ft = load_dataset("scrapegraphai/scrapegraph-100k-finetuning")
print(ft["human_eval"][0])

The modelling/metrics.py module provides evaluation functions for JSON extraction tasks.

Check if a JSON string is valid and complies with a schema:

from modelling.metrics import json_validator

schema = {
    "type": "object",
    "properties": {"name": {"type": "string"}},
    "required": ["name"]
}

result = json_validator('{"name": "John"}', schema)
# {'is_valid': True, 'is_compliant': True}

Evaluate extraction quality with magic_metric:

from modelling.metrics import magic_metric, parse_json_remove_duplicates

pred = '{"name": "John", "age": 30, "tags": ["a", "b"]}'
true = {"name": "john", "age": 30, "tags": ["b", "a"], "city": "NYC"}

pred = parse_json_remove_duplicates(pred)
if pred is None:
    pred = {}
result = magic_metric(pred, true)

Returns:

Requires Python 3.12+ and uv.

uv sync --extra dev

For local serving with vllm (installed separately due to conflicting numba/numpy pins):

uv pip install vllm

• 🤗 Hugging Face Dataset
• 🤗 Hugging Face Dataset for Finetuning
• 🤗 Finetuned Model (sgai-qwen3-1.7b)
• 📄 arXiv Paper
• 🕷️ ScrapeGraphAI Library

modal run modelling/train.py --dry-run
modal run --detach modelling/train.py

modal run modelling/evaluation.py
modal run --detach modelling/evaluation.py
modal run --detach modelling/evaluation.py --lora /checkpoints/default/final

Merge the LoRA adapter into the base model for faster inference (no runtime LoRA overhead, enables CUDA graphs):

python -m modelling.merge_lora \
  --model-name Qwen/Qwen3-1.7B \
  --lora-path sg-checkpoints/efficient-frost-76/final

Saves to sg-checkpoints/efficient-frost-76/merged by default. Use --output-path to override.

AWQ W4A16 (calibrated on the finetuning train split via llmcompressor):

modal run modelling/convert_awq.py --lora-path /checkpoints/efficient-frost-76/final
modal run modelling/convert_awq.py --model-path /checkpoints/efficient-frost-76/merged --push-to-hub

GGUF (llama.cpp, default quants q4_k_m,q8_0):

modal run modelling/convert_gguf.py --lora-path /checkpoints/efficient-frost-76/final
modal run modelling/convert_gguf.py --model-path /checkpoints/efficient-frost-76/merged --quant q4_k_m --push-to-hub

Quantized model (recommended, fastest):

vllm serve sg-checkpoints/efficient-frost-76/merged-w4a16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --port 8000

Merged model (fp16):

vllm serve sg-checkpoints/efficient-frost-76/merged \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --port 8000

On a 3090 (24GB), use the quantized model with Marlin kernels:

vllm serve sg-checkpoints/efficient-frost-76/merged-w4a16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.8 \
  --enable-prefix-caching \
  --port 8000

Or fp16 with --enforce-eager to skip CUDA graph capture:

vllm serve sg-checkpoints/efficient-frost-76/merged \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enforce-eager \
  --port 8000

With LoRA (without merging, slowest):

vllm serve Qwen/Qwen3-1.7B \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-lora \
  --lora-modules adapter=sg-checkpoints/efficient-frost-76/final \
  --enforce-eager \
  --port 8000

Then:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sg-checkpoints/efficient-frost-76/merged-w4a16",
    "messages": [{"role": "user", "content": "your prompt here"}],
    "temperature": 0.0,
    "max_tokens": 4096,
    "repetition_penalty": 1.1,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

@misc{brach2026scrapegraphai100kdatasetschemaconstrainedllm,
      title={ScrapeGraphAI-100k: Dataset for Schema-Constrained LLM Generation},
      author={William Brach and Francesco Zuppichini and Marco Vinciguerra and Lorenzo Padoan},
      year={2026},
      eprint={2602.15189},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2602.15189},
}