You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This directory contains a set of scripts to evaluate the performance of different presses on different datasets. We currently support the following datasets:
- Results (predictions & metrics) are saved in the `results` directory.
28
-
- All available presses are listed in the `PRESS_DICT` variable in `evaluate.py`.
29
-
- Additional arguments are --device, --fraction, --max_new_tokens, --max_context_length and --compress_questions. For more information, run `python evaluate.py --help`
30
-
- Finally we also provide a bash script `evaluate.sh` to facilitate the evaluation of multiple presses (1 per GPU) with different compression ratios.
31
-
32
-
33
-
## Benchmarks
34
-
35
-
We provide benchmark results from 7 presses and 3 models. We include a variant of SnapKV where we include the question in the compression process as in the original paper (snapkv w/ question). All performance curves can be found in the [assets](assets) directory, and predictions are available [here](https://drive.google.com/drive/folders/14BilGw07v8tOUUct-5nDhQlN3zIX9BUf?usp=drive_link).
36
-
37
-
<details><summary>
38
-
39
-
### RULER
40
-
</summary>
41
-
42
-
Average performance the 13 tasks of the RULER dataset with 4k context length (per task results [here](../evaluation/assets/)):
- snapkv w/ question consistently outperforms other methods. However this method can't be use for use cases such as prompt caching as it requires the question to be known beforehand.
48
-
- All presses show degradation in performance even for small compression ratios.
49
-
- llama3.1-8b-instruct is more robust to compression than other models and expected attention performs better than others.
50
-
- mistral-nemo-instruct-2407 is more robust to random pruning than other models.
51
-
- For phi-3.5-mini and mistral-nemo-instruct-2407, all presses perform poorly compared to baseline presses such as random (remove KV pairs randomly) or streaming llm (remove the middle KV pairs). This is especially true for the subtask [niah_single_3](assets/ruler_4096_niah_single_3.png) where most presses fail to perform a proper copy-paste of a long needle in a haystack. This might be related to [induction heads](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)
52
-
- For phi-3.5-mini, we ran an additional experiment with a different compression ratio per layer (as in [this notebook](../notebooks/per_layer_compression_demo.ipynb)) which largely outperformed it's uniform compression counterpart (see purple cross on 2nd plot). The ratios where determined by grid search on 20/6500 samples from RULER (so results can be questionable).
- Metrics are adapted from loogle benchmark, see [here](../evaluation/loogle/calculate_metrics.py). The plot show the average score (mean over all submetrics) for each task.
70
-
- The metrics are not always correlated with the quality of the answer, especially for longdep_qa task. LLM-as-a-judge may better suited for a more refined evaluation.
71
-
- Again, snapkv w/ question consistently outperforms other methods.
72
-
- In longdep_qa, the model looses track on counting (e.g. answer to "How many times is person x mentioned?" gets lower with increased compression ratio). This is not necessarily reflected in the metrics.
73
-
- Llama3.1-8b-instruct seems to be more robust to compression.
74
-
- Observed attention context had to be truncated at 10 000 tokens to prevent OOM issues, as the attention matrix needs to be materialized.
75
-
- For shortdep_cloze task, the output formatting is often ignored leading to performance degradation even for low compression ratios. Interestingly, the model may still be able to answer the question correctly.
76
-
- mistral-nemo-instruct-2407 fails to perform well on the shortdep_cloze task, even without compression, and is thus not reported.
77
-
- shortdep_cloze task runs OOM for phi-3.5-mini at compression ratio 0.0 and is thus missing.
28
+
### Configuration
78
29
79
-
</details>
30
+
Customize your evaluation by editing `evaluate_config.yaml`. This allows you to flexibly configure a variety of settings, like the `fraction` of dataset to use (for quick testing) and the model arguments (e.g. for scaling RoPE). For complete parameter details, see the `evaluation_config.yaml`
80
31
81
-
<details><summary>
82
32
83
-
### Infinitebench
84
-
</summary>
33
+
### Available Presses and Datasets
34
+
We support evaluation with all the presses implemented in the library (and possible combinations).
- All implemented presses are listed in the `PRESS_REGISTRY` variable in `evaluate_registry.py`.
37
+
- All implemented dataset are listed in `DATASET_REGISTRY` variable in `evaluate_registry.py`.
94
38
39
+
At the moment, we support the following standard popular benchmarks:
95
40
96
-
Observations:
97
-
- All task where run with max_len=70_000 tokens, except for observed attention which used 10_000 tokens.
98
-
- For kv-retrieval subtask, streaming LLM (keep last N tokens) performs better than other methods. While this may be surprising at first, respecting the format of the task `(Extract the value corresponding to the specified key in the JSON object below. JSON data: {"7de93460-b65f-404e-9a7d-af2da2c8abb5": "2d9ab7c8-394a-4062-9928-310e39201a2f", ...}. Key: "70d1b207-d1e8-4591-95b8-9c85aceb8956"`
99
-
helps to understand this behavior. The information is homogeneously distributed in the context, and any token could potentially be relevant for answering the question. Streaming LLM will have access to all last tokens, while other methods will potentially create "holes".
100
-
- Mistral-nemo-instruct-2407 performs poorly on kv-retrieval subtask compared to other models and is thus excluded from the plots.
101
-
- For longbook-choice-eng, many compression methods are able to obtain good compression ratios. Thus, longbook-choice-eng is an example of a task that can be compressed effectively.
102
-
- For longbook-qa-eng, expected attention and snapkv perform better than other methods (note the performance difference of llama3.1-8b-instruct and phi3.5/mistral-nemo).
103
-
- For longdialogue-qa-eng, there's an interesting crossover between different compression methods. For higher compression, snapkv performs relatively well across models.
📚 **For detailed information** about each dataset or implementing custom benchmarks, see the individual README files in the benchmarks directory.
106
49
107
50
108
-
### Conclusions
51
+
### Multi GPU Evaluation
52
+
Use the provided `evaluate.sh` script to run multiple presses simultaneously across different GPUs with varying compression ratios.
109
53
54
+
### Discussion
110
55
The methods benchmarked so far are not able to efficiently compress the KV cache while maintaining performance on several long-context datasets and models.
111
56
In particular, exact information retrieval tasks such as kv-retrieval are challenging for the current methods.
112
57
Further methods could be explored:
@@ -116,24 +61,4 @@ Further methods could be explored:
116
61
- Move beyond pruning, as this method is fundamentally limited (see last figure in [this notebook](../notebooks/expected_attention.ipynb))
117
62
- Fine-tuning LLMs to deal with compressed KV caches
118
63
119
-
We encourage contributions to explore these ideas and improve the performance of long-context LLMs with compressed caches.
120
-
121
-
## How to add a dataset
122
-
123
-
Each dataset directory is structured as follows:
124
-
125
-
```bash
126
-
$dataset
127
-
├── README.md
128
-
├── calculate_metrics.py
129
-
├── create_huggingface_dataset.py
130
-
```
131
-
132
-
Where:
133
-
-`create_huggingface_dataset.py` is a script that generates the Hugging Face dataset from the original dataset. Each dataset is associated with a set of parquet files with the following structure:
134
-
-`context`: ...
135
-
-`question`: ...
136
-
-`answer_prefix`: ...
137
-
-`answer`: ...
138
-
-`max_new_tokens`: ...
139
-
-`calculate_metrics.py` is a script that calculates the metrics based on the output of `evaluate.py`
64
+
We encourage contributions to explore these ideas and improve the performance of long-context LLMs with compressed caches. We provide benchmark results from 7 presses and 3 models. We include a variant of SnapKV where we include the question in the compression process as in the original paper (snapkv w/ question). All performance curves can be found in the [assets](assets) directory, and predictions are available [here](https://drive.google.com/drive/folders/14BilGw07v8tOUUct-5nDhQlN3zIX9BUf?usp=drive_link).
- snapkv w/ question consistently outperforms other methods. However this method can't be use for use cases such as prompt caching as it requires the question to be known beforehand.
24
+
- All presses show degradation in performance even for small compression ratios.
25
+
- llama3.1-8b-instruct is more robust to compression than other models and expected attention performs better than others.
26
+
- mistral-nemo-instruct-2407 is more robust to random pruning than other models.
27
+
- For phi-3.5-mini and mistral-nemo-instruct-2407, all presses perform poorly compared to baseline presses such as random (remove KV pairs randomly) or streaming llm (remove the middle KV pairs). This is especially true for the subtask [niah_single_3](assets/ruler_4096_niah_single_3.png) where most presses fail to perform a proper copy-paste of a long needle in a haystack. This might be related to [induction heads](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)
28
+
- For phi-3.5-mini, we ran an additional experiment with a different compression ratio per layer (as in [this notebook](../notebooks/per_layer_compression_demo.ipynb)) which largely outperformed it's uniform compression counterpart (see purple cross on 2nd plot). The ratios where determined by grid search on 20/6500 samples from RULER (so results can be questionable).
- Metrics are adapted from loogle benchmark, see [here](../evaluation/loogle/calculate_metrics.py). The plot show the average score (mean over all submetrics) for each task.
46
+
- The metrics are not always correlated with the quality of the answer, especially for longdep_qa task. LLM-as-a-judge may better suited for a more refined evaluation.
47
+
- Again, snapkv w/ question consistently outperforms other methods.
48
+
- In longdep_qa, the model looses track on counting (e.g. answer to "How many times is person x mentioned?" gets lower with increased compression ratio). This is not necessarily reflected in the metrics.
49
+
- Llama3.1-8b-instruct seems to be more robust to compression.
50
+
- Observed attention context had to be truncated at 10 000 tokens to prevent OOM issues, as the attention matrix needs to be materialized.
51
+
- For shortdep_cloze task, the output formatting is often ignored leading to performance degradation even for low compression ratios. Interestingly, the model may still be able to answer the question correctly.
52
+
- mistral-nemo-instruct-2407 fails to perform well on the shortdep_cloze task, even without compression, and is thus not reported.
53
+
- shortdep_cloze task runs OOM for phi-3.5-mini at compression ratio 0.0 and is thus missing.
- All task where run with max_len=70_000 tokens, except for observed attention which used 10_000 tokens.
74
+
- For kv-retrieval subtask, streaming LLM (keep last N tokens) performs better than other methods. While this may be surprising at first, respecting the format of the task `(Extract the value corresponding to the specified key in the JSON object below. JSON data: {"7de93460-b65f-404e-9a7d-af2da2c8abb5": "2d9ab7c8-394a-4062-9928-310e39201a2f", ...}. Key: "70d1b207-d1e8-4591-95b8-9c85aceb8956"`
75
+
helps to understand this behavior. The information is homogeneously distributed in the context, and any token could potentially be relevant for answering the question. Streaming LLM will have access to all last tokens, while other methods will potentially create "holes".
76
+
- Mistral-nemo-instruct-2407 performs poorly on kv-retrieval subtask compared to other models and is thus excluded from the plots.
77
+
- For longbook-choice-eng, many compression methods are able to obtain good compression ratios. Thus, longbook-choice-eng is an example of a task that can be compressed effectively.
78
+
- For longbook-qa-eng, expected attention and snapkv perform better than other methods (note the performance difference of llama3.1-8b-instruct and phi3.5/mistral-nemo).
79
+
- For longdialogue-qa-eng, there's an interesting crossover between different compression methods. For higher compression, snapkv performs relatively well across models.
80
+
81
+
</details>
82
+
83
+
84
+
## How to add a dataset
85
+
86
+
Each dataset directory is structured as follows:
87
+
88
+
```bash
89
+
$dataset
90
+
├── README.md
91
+
├── calculate_metrics.py
92
+
├── create_huggingface_dataset.py
93
+
```
94
+
95
+
Where:
96
+
-`create_huggingface_dataset.py` is a script that generates the Hugging Face dataset from the original dataset. Each dataset is associated with a set of parquet files with the following structure:
97
+
-`context`: ...
98
+
-`question`: ...
99
+
-`answer_prefix`: ...
100
+
-`answer`: ...
101
+
-`max_new_tokens`: ...
102
+
-`calculate_metrics.py` is a script that calculates the metrics based on the output of `evaluate.py`
0 commit comments