SHALE is a Scalable HALlucination Evaluation benchmark built on an automated data construction pipeline. It is designed to assess both faithfulness and factuality hallucinations via a fine-grained hallucination categorization scheme. SHALE comprises over 30K image-instruction pairs spanning 12 representative visual perception aspects for faithfulness and 6 knowledge domains for factuality, considering both clean and noisy scenarios.

The evaluation data can be downloaded from this LINK.

The evaluation data is structured in JSON format, as follows:

[
  {
    "id": 1,
    "task": "PosYNQ",
    "hallucination_type": "faithfulness",
    "evaluation_dimension": "type",
    "entity_type": "object",
    "images": [
      "./Image/1.jpg"
    ],
    "instruction": "Is the object in the image a pan?",
    "ground_truth": "Yes"
  },
    ...
]

The images are saved in ./Image, and the instructions are saved in ./Query.

The code for model evaluation is saved in ./Evaluation.