Add Janus-1.3B

[hills-code]

We add Janus-1.3B model to reproduce the results in paper.

	MME	MMB (w/o circular)	SEED	MMMU_DEV_VAL	MM-Vet	POPE
VLMEvalkit (Reproduce)	1342.4	69.8	63.8	31.2	36.8	85.5 (overall) 87.1 (random)
Paper	1338.0	69.4	63.7	30.5	34.3	87 (random)

You can run the evaluation with the following code:

torchrun --nproc-per-node=8 run.py --data POPE MMMU_DEV_VAL MMBench_DEV_EN MME SEEDBench_IMG MMVet --model janus_1.3b --verbose

Note:

• We evaluate MMBench without circular mode. You should set circular=False in the file vlmeval/dataset/image_mcq.py
• We use the official evaluation of MM-Vet with GPT-4 evaluator.

[wusize]

We add Janus-1.3B model to reproduce the results in paper.

MME MMB
(w/o circular) SEED MMMU_DEV_VAL MM-Vet POPE
VLMEvalkit (Reproduce) 1342.4 69.8 63.8 31.2 36.8 85.5 (overall)
87.1 (random)
Paper 1338.0 69.4 63.7 30.5 34.3 87 (random)
You can run the evaluation with the following code:

torchrun --nproc-per-node=8 run.py --data POPE MMMU_DEV_VAL MMBench_DEV_EN MME SEEDBench_IMG MMVet --model janus_1.3b --verbose

Note:

• We evaluate MMBench without circular mode. You should set circular=False in the file vlmeval/dataset/image_mcq.py
• We use the official evaluation of MM-Vet with GPT-4 evaluator.

Hi! There are two splits in MMMU_DEV_VAL, i.e., dev and validation. May I know which one the numbers in the table correspond to?

[charlesCXK]

We add Janus-1.3B model to reproduce the results in paper.
MME MMB
(w/o circular) SEED MMMU_DEV_VAL MM-Vet POPE
VLMEvalkit (Reproduce) 1342.4 69.8 63.8 31.2 36.8 85.5 (overall)
87.1 (random)
Paper 1338.0 69.4 63.7 30.5 34.3 87 (random)
You can run the evaluation with the following code:

torchrun --nproc-per-node=8 run.py --data POPE MMMU_DEV_VAL MMBench_DEV_EN MME SEEDBench_IMG MMVet --model janus_1.3b --verbose

Note:

• We evaluate MMBench without circular mode. You should set circular=False in the file vlmeval/dataset/image_mcq.py
• We use the official evaluation of MM-Vet with GPT-4 evaluator.

Hi! There are two splits in MMMU_DEV_VAL, i.e., dev and validation. May I know which one the numbers in the table correspond to?

Hi, it was VAL set~