Inconstancy between predict and predict_proba when using strings labels

[AsafYehudai]

I attempted to work with the probabilities obtained from predict_proba. For a sanity check, I used argmax and id2label to retrieve the label name, but encountered different results. After some trial and error, I realized that the argmax values were indeed the same. Upon further investigation, I found that the order of the labels used for probabilities is based on the sorted labels, which may differ from the order in id2label. Consequently, in the case of the "negative" and "positive" examples from SST2, they only coincidentally work because "negative" comes before "positive" and is mapped to 0, not 1.

Another point to consider is modifying predict_proba to also return predictions. Although it performs quickly, having it include predictions would eliminate the need to call prediction functions separately.

I appreciate your efforts.

Best regards,
Asaf

[tomaarsen]

Indeed, the labels have to match the order of the integer labels if those were used during training. Consider the labels list as an id2label: labels[0] must be the string label for the integer label 0, label[1] must be the string label for the integer label 1, etc.

For example, in my model here, the order of the two labels is no coincidence. If the label order is changed, the outputs will be wrong. Does that make sense?

I see that this may have not been clear. Perhaps I can clarify some documentation further. As for the predict_proba functionality, the reason that I don't also return predictions there is because I want to stick to the scikit-learn inspired behaviour where classifiers have predict and predict_proba methods with specific output shapes.

• Tom Aarsen

[AsafYehudai]

I'm unsure if I fully grasp the first part of your explanation. In my scenario, I trained using a list of labels where their order dictated the id2label, that is labels[0] corresponded to id 0. However, I noticed an anomaly in how the probabilities are assigned. While labels[0] represents id 0, probs[0] doesn't necessarily reflect the score for labels[0]; instead, it seems to correspond to the score for labels.sort()[0]. I reached this conclusion when comparing the argmax of probs (from predict_proba) with the label obtained from predict; they didn't align.

In your example with only two classes, "negative" and "positive," predict and predict_proba behaved as expected because "negative" comes before "positive" lexicographically. However, when dealing with a more extensive set of classes (60+), I observed consistency only after sorting the labels before providing them to the model. If this interpretation makes sense, a potential solution could involve incorporating label sorting into your code. Users should be informed to rely on id2label for accurate results rather than the order of the labels they provide. There may be other solutions worth exploring as well.

Thanks,
Asaf

[dennis-zyska]

I have experienced the same problem, here is an example code to show the issue:

from datasets import DatasetDict
from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset
import pandas as pd
from datasets import Dataset, DatasetDict
from tqdm import tqdm as tqdm

data = [
    {'sentence': "Test", 'label': "Test"},
    {'sentence': "Test2", 'label': "Test"},
    {'sentence': "Test3", 'label': "Test"},
    {'sentence': "Test4", 'label': "Test"},
    {'sentence': "1", 'label': "Number"},
    {'sentence': "2", 'label': "Number"},
    {'sentence': "3", 'label': "Number"},
    {'sentence': "4", 'label': "Number"},
    {'sentence': "§", 'label': "Symbol"},
    {'sentence': "$", 'label': "Symbol"},
    {'sentence': "%", 'label': "Symbol"},
    {'sentence': "&", 'label': "Symbol"},
    {'sentence': "you", 'label': "Stop"},
    {'sentence': "our", 'label': "Stop"},
    {'sentence': "we", 'label': "Stop"},
    {'sentence': "I", 'label': "Stop"},
]

training_data = pd.DataFrame(data)
train_dataset = Dataset.from_pandas(training_data)
val_dataset = Dataset.from_pandas(training_data)
test_dataset = Dataset.from_pandas(training_data)  

model = SetFitModel.from_pretrained(
    #"sentence-transformers/paraphrase-mpnet-base-v2",
    "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",
    labels=["Test", "Number", "Symbol", "Stop"]
)

args = TrainingArguments(
    batch_size=8,
    num_epochs=4,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    metric="accuracy",
    column_mapping={"sentence": "text", "label": "label"}  # Map dataset columns to text/label expected by trainer
)

trainer.train()

print("Build dataframe...")
results = []
for i in tqdm(range(len(data))):
    results.append((data[i]['sentence'], model.predict(data[i]['sentence']), *(float(x) for x in model.predict_proba(data[i]['sentence']))))
results = pd.DataFrame(results, columns =['sentence','predicted_label', *model.labels])
print(results)

Output:

 	sentence 	predicted_label 	Test 	Number 	Symbol 	Stop
0 	Test 	Test 	0.017231 	0.020279 	0.017347 	0.945143
1 	Test2 	Test 	0.017876 	0.019482 	0.015799 	0.946843
2 	Test3 	Test 	0.018207 	0.019819 	0.015734 	0.946240
3 	Test4 	Test 	0.018152 	0.019836 	0.015739 	0.946272
4 	1 	Number 	0.940297 	0.022526 	0.019100 	0.018077
5 	2 	Number 	0.941060 	0.022865 	0.018269 	0.017806
6 	3 	Number 	0.941635 	0.022680 	0.018018 	0.017666
7 	4 	Number 	0.940461 	0.022182 	0.019324 	0.018033
8 	§ 	Symbol 	0.016878 	0.019262 	0.948708 	0.015152
9 	$ 	Symbol 	0.021275 	0.024838 	0.935522 	0.018365
10 	% 	Symbol 	0.019616 	0.021292 	0.942882 	0.016209
11 	& 	Symbol 	0.017099 	0.018344 	0.949454 	0.015103
12 	you 	Stop 	0.023334 	0.934895 	0.020981 	0.020790
13 	our 	Stop 	0.023235 	0.933926 	0.022215 	0.020624
14 	we 	Stop 	0.021815 	0.939222 	0.019673 	0.019290
15 	I 	Stop 	0.021573 	0.939438 	0.020385 	0.018604