I can't find the parameter Cs in code

[LouisDRR]

Hello author, could you please explain why I cannot find any explicit definition of the spatial head parameter Cs​ in the code?

[andy-yang-1]

The parameter is translated to sparsity ratio, see code

[LouisDRR]

I still don't understand. The two types of masks generated by the ‘’get_attention_mask‘’ function seem to be used only for online prefilling. The ‘’sparsity‘’parameter is used to generate the block mask for FlexAttention. Suppose we set the parameter ‘’sparsity‘’ to 0.25 and obtain the corresponding ‘’width_frame‘’ via the function ‘’sparsity_to_width’‘. How should this be converted into the spatial head parameter Cs and temporal head parameter Ct for FLOPS calculation? I would be very grateful if you could answer this question.

[haochengxi]

#13 (comment)

Please refer to here for more details. Width_frame directly indicates the width of the sparse attention pattern in the spatial head. Temporal Head's sparsity pattern is the same as the spatial head after the Layout Transformation defined in our paper.

[LouisDRR]

#13 (comment)

Please refer to here for more details. Width_frame directly indicates the width of the sparse attention pattern in the spatial head. Temporal Head's sparsity pattern is the same as the spatial head after the Layout Transformation defined in our paper.

Thank you！