Question about the mask token within the prompt area.

[ziz-797]

Hi,

Thank you very much for your excellent and inspiring work!

I have a question regarding the mask token design during training. I noticed that some structural tokens within the prompt region (e.g., , ) are also masked, and their logits are set to -inf during inference.

I was wondering:

What is the main motivation behind masking these structural tokens?

Does this strategy contribute to improved model performance or training stability?

Is this design primarily intended to enforce strict generation constraints, or does it also provide benefits during representation learning?

I would greatly appreciate any clarification on the rationale behind this design choice.

Thank you very much for your time and support.

Best regards,
ziz-797

[yyyouy]

Thanks for the question.

This design mainly follows LLaVA-OneVision. The motivation is to let the model focus more on the content rather than the fixed structural tokens in the prompt template.

In addition, masking these structural tokens and setting their logits to -inf during inference helps enforce the predefined generation structure more strictly.

However, we have not performed a dedicated ablation study on this design, so we cannot conclude whether it improves model performance or training stability.

As for representation learning, we did not specifically study this aspect, so we are not sure whether it provides additional benefits there.

Thanks again for pointing this out.

[ziz-797]

OK, Thank you very much!