Integrate Q-Filters

[NathanGodey]

PR description

Description of your PR. Fixes # (issue) (if applicable)

New press checklist (if applicable)

• I added mypress_press.py in the presses directory
• I added MyPress in __init__.py
• I updated the README.md with a 1 liner about my new press in the Available presses section
• I added my press in the default_presses list in tests/default_presses.py

[SimJeg]

Hi, thanks for contributing to kvpress ! Could you resolve conflicts with the main branch so that I can run the CI/CD ? I will add a few comments. Please don't forget to sign off your commits using git commit -s

[SimJeg]

@NathanGodey in addition to comments above, you'll need to sign off your past commits for the DCO:

To add your Signed-off-by line to every commit in this branch:
Ensure you have a local copy of your branch by checking out the pull request locally via command line.
In your local branch, run: git rebase HEAD~2 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin qfilters

[NathanGodey]

I'm terrible at git so the rebasing was a bit messy... It should work now, let me know if I can help on anything :)

[SimJeg]

@NathanGodey my review was pending since last week, I forgot to publish it 🤦
The DCO now works thank you, don't forget to sign-off your next commits !

[SimJeg]

@NathanGodey thanks for all the updates ! I asked for 2 3 4 minor updates, once it's done we're ready to merge, congrats 👏

[SimJeg]

@NathanGodey could you push the 4 updates I mentioned above ?

For info I ran an experiment where I saved the mean and covariance for ExpectedAttention using 100 samples from the BookSum dataset. For fairness in comparison I also removed the value norm rescaling. With 50% compression on RULER I get:

• Qfilters: 62.82
• Expected Attention with on the fly statistics: 73.41
• Expected Attention with saved statistics: 76.40

So saving statistics makes it both better and faster !

[NathanGodey]

That sounds great! It seems to confirm that this "unimodal" filtering is inherent to the model and not particularly context-dependent.

Now, I'm wondering what explains the gap between Q-Filters and pre-computed ExpectedAttention, because they start looking more and more the same (maybe covariance? handling of positional encoding? forced sink tokens?); I would also like to see if the saved stats help for larger compression ratios, where Q-Filters seems more effective compared to ExpectedAttention. I'll run the experiments when/if I have some time, let me know if you want to stay in the loop (maybe elsewhere :) ).

[SimJeg]

what explains the gap between Q-Filters and pre-computed ExpectedAttention

I would say:

• apply future RoPE rotation to saved query: I get 79.75 -> 74.56 if I remove this rotation (with saved statistics + vnorm)
• better estimation of E[exp(<Q, K>)], partly because of covariance (with mean only I get 71.86)

[SimJeg]

@NathanGodey you still have 1 comment to address (not sure you received notification in the thread)