Problem: Object hallucination in LVLMs—generating descriptions of non-existent objects worsens as generation length increases.

Root Cause: Models over-rely on language priors rather than visual evidence.

Solution: A training-free Self-Validation Framework with two steps:

1. Candidates Verifications: Sample multiple candidate captions, extract candidate objects, and verify each object's existence via Language-Prior Free Verfication (LPFV).

2. Final Caption Production:

   • Best-of-N (BoN): Select the caption with highest verification confidence
   • Filter-then-Aggregate (FtA): Filter low-confidence objects and aggregate into final caption

Results: 65.6% improvement on CHAIR_I with LLaVA-v1.5-7B, surpassing prior SOTA methods.

1. Environment Setup
2. Data Preparation
3. Re-implementation
4. CHAIR Evaluation
5. Acknowledgement

conda create -yn selfval python=3.10
conda activate selfval
pip install -r requirements.txt

Following less-is-more project. The test set used in our paper for CHAIR evaluation is provided in ./chair500/chair-500.jsonl. The data is randomly sampled from the MSCOCO validation set with a random seed of 0.

To collect test set images, you can open an OneDrive link to download the 500 images.

To collect the corresponding object ground truth (annotations) data: http://images.cocodataset.org/annotations/annotations_trainval2014.zip

Then, you can use our proposed Best-of-N Selection (BoN) or Filter-then-Aggregate (FtA) methods through completing these arguments:

# For BoN
"$PYTHON_PATH" -m llava.eval.selection \
    --model-path $MODEL_PATH \
    --question-file $QUESTION_FILE \
    --answers-file $OUTPUT_PATH \
    --sample-num $SAMPLE_NUM \
    --extract-method mscoco \
    --cache $CACHE_PATH \
    --image-dir $IMAGEDIR


# For FtA
python -m llava.eval.aggreagate \
    --model-path $MODEL_PATH \
    --question-file $QUESTION_FILE \
    --answers-file $OUTPUT_PATH \
    --sample_num $SAMPLE_NUM \
    --extract-method $EXTRACTE_METHOD \
    --cache $CACHE_PATH \
    --image-dir $IMAGEDIR

MODEL_PATH is llava model path.

QUESTION_FILE is the path of chair-500.jsonl, as the model input.

OUTPUT_PATH is the output path of result path.

SAMPLE_NUM is the sampling times.

EXTRACTE_METHOD can be chosen from ['mscoco', 'self', 'statis'].

CACHE_PATH is the path of chair.pkl, positioned in chair500/chair.pkl.

IMAGEDIR is the image dir of chair-500.

For Best-of-N Selection (BoN), run scripts/selection.sh. For Filter-then-Aggregate (FtA), run scripts/aggregation.sh.

We have provided CHAIR evaluation metric in eval_chair500.sh.

This repo is built on LLaVA (models), VCD and Less-is-more (CHAIR evaluation). Many thanks for their efforts. The use of our code should also follow the original licenses.