Question : evaluation after every training epoch

[Yongtae723]

Thank you

Hello!
I am Yongtae, a senior ML engineer in japan.
Thank you for publishing a genuinely excellent paper and code.
Few-shot learning and multilingual support are appreciated by engineers like me who work abroad!

Question

I felt this model easily overfit to train data if the number of epochs is over 2 or train data contains similar data.
Therefore I would like to evaluate the model after every training epoch to find out the best epoch number.

But as shown here, it seems difficult to evaluate the model at every epoch, because the body part is trained on full epoch at the beginning of the training.

so I would like to change like below

for epoch in num_epochs:
    self.model.model_body.fit(
                train_objectives=[(train_dataloader, train_loss)],
                epochs=1,
                steps_per_epoch=train_steps,
                optimizer_params={"lr": learning_rate},
                warmup_steps=warmup_steps,
                show_progress_bar=True,
                use_amp=self.use_amp,
            )

    if not is_differentiable_head or not self._freeze:
            # Train the final classifier
            self.model.fit(
                x_train,
                y_train,
                num_epochs=1,
                batch_size=batch_size,
                learning_rate=learning_rate,
                body_learning_rate=body_learning_rate,
                l2_weight=l2_weight,
                show_progress_bar=True,
            )

    somehow_evaluete()

Does it make sense to you?
Or if I fork and make that change, are there any problem?

I am looking forward to your reply

Best and thank you in advance!

[tomaarsen]

Hello!

My understanding of the paper (as another enthusiast like yourself) is that the model is trained in two separate phases. i.e.:

1. The SentenceTransformer body is fully finetuned.
2. Only then, the classifier head is fully trained.

The approach that you're describing in your code block is to repeatedly finetune one epoch on the SentenceTransformer body and then train one epoch on the classifier head, which is not equivalent to the paper.

The best that may be implementable is a (list of) callback function(s) that can be provided via some argument to SetFitTrainer, which may be called after every epoch of either of the two training phases, à la transformer or PyTorch Lightning callbacks. However, this is a bit awkward due to the two separate training phases.

• Tom Aarsen

[Yongtae723]

Hi Tom!
Thank you for your reply!

I understand your point. However, it is still very difficult to evaluate the performance of the model at each epoch when training SentenceTransformer (since the model_head is not trained).

However, I have a good idea to solve this problem and am now implementing the training of setfit with PyTorch Lightning.
I am almost done with the coding and will contact you after it is published, so please give us a brief comment if possible.

I also expect that wrap in PyTorch Lighining will facilitate logging, use of callbacks, and management of training devices.

[Yongtae723]

Hi Tom!
I am so sorry for the late reply.
We finally finished coding and published here.

Regarding evaluation after every epoch, See here

if you have any questions or opinions, feel free to give me a comment.
Also if you feel our implementation is good enough, could you introduce our repository in your README?

thank you.
I hope you doing great

Yongtae

[tomaarsen]

Very interesting work! Well done. It certainly solves some problems that we currently have with the main work (callbacks, gradient accumulation, etc.). I'm open to mentioning the work in a "Related work" section of sorts in the README, but I'll have to leave that decision to @lewtun.

• Tom Aarsen

[Yongtae723]

Hi Tom!

Can I make PR to add mentioning to introduce our work in README?

Of course, I understand The final decision will be made by @lewtun.
Thank you

[tomaarsen]

As far as I'm concerned: Feel free!