[HELP WANTED]Tutorial for nn.Transformer with nn.TransformerDecoder

[xdwang0726]

Hi,
The existing tutorials for nn.Transformer e.g. https://pytorch.org/tutorials/beginner/transformer_tutorial.html and https://github.com/pytorch/examples/tree/master/word_language_model both are only use nn.TransformerEncoder. There is no tutorial using nn.TransformerDecoder. I think it is better to add one example that include nn.TransformerDecoder so that users can easy get started with. As for my own experience, I have some troubles using nn.TransfomerDecoder in inference process.

[zhangguanheng66]

Could you propose a problem for TransformerDecoder. Based on my experience, the applications of nn.TransformerDecoder should be very similar to that of nn.TransformerEncoder.

[xdwang0726]

Could you propose a problem for TransformerDecoder. Based on my experience, the applications of nn.TransformerDecoder should be very similar to that of nn.TransformerEncoder.

Thank you for your reply! I have some trouble in nn.TransformerDecoder's inference process. I am using transformer for summarization task. My training goes well (got a smoothy converged loss), but the inference process has some problems (i.e. always generate the same token when inference). I have followed the official tutorial, but didn't really help. I am wondering is there any resource I can refer to? Thank you!

[zhangguanheng66]

@xdwang0726 Did you mask your target and memory inputs to TransformerDecoder properly?

[zhangguanheng66]

BTW, if you would like to publish your work on pytorch/tutorial, I'm happy to take a look.

[xdwang0726]

Thank you for your reply!
I used generate_square_mask to obtain the tgt_mask, but for memory_mask, I have some problem with the dimensions as generate_square_mask can only generate square masks which is not fit the memory_mask's dimension. So I set memory_mask = None.

[xdwang0726]

BTW, if you would like to publish your work on pytorch/tutorial, I'm happy to take a look.

Here attached my codes. Thank you for your kind help!
Forward steps:

    encoder_vec = self.bert_encoder(src_input_ids, src_token_type_ids, src_attention_mask)
    tgt_mask = self.generate_square_subsequent_mask(tgt_input_ids.shape[1]).to(self.device)  # tgt_mask: (T, T)
    #memory_mask = self.generate_subsequent_mask(len(encoder_vec), tgt_input_ids.shape[1]).to(self.device)  # memory_mask: (T,S)
    decoder_outputs = self.bert_decoder(encoder_vec, tgt_input_ids, tgt_mask, inference)
    score, lsm_score = self.generator(decoder_outputs)
    score = score.transpose(0, 1).squeeze(0)  # shape [num_tgt_tokens, num_classes]

Inference Steps:

    with torch.no_grad():
        batch_size = 1

        # encoder
        encoder_vec = self.bert_encoder(src_input_ids, src_token_type_ids, src_attention_mask)
        # create initial tokens: shape tensor([[101]])
        generated_seq = torch.full((batch_size, 1), Constants.BOS, dtype=torch.long, device=self.device)
        generated_seq4 = tgt_input_ids[:, 0].unsqueeze(0)

        for i in range(1, max_tgt_seq_len):
            tgt_mask = self.generate_square_subsequent_mask(generated_seq.shape[1]).to(self.device)
            decoder_outputs = self.bert_decoder(encoder_vec, generated_seq, tgt_mask, inference)

            _, lsm_score = self.generator(decoder_outputs)
            # Take token with largest probability and use it for the next words
            generated_token_index = torch.topk(lsm_score, 1, dim=-1)[1][-1, :, :]  # lsm_score shape [num_tokens, bs, vocab_size]

            # Concatenate generated token with sequence
            generated_seq = torch.cat((generated_seq, generated_token_index), dim=-1)

[zhangguanheng66]

Thank you for your reply!
I used generate_square_mask to obtain the tgt_mask, but for memory_mask, I have some problem with the dimensions as generate_square_mask can only generate square masks which is not fit the memory_mask's dimension. So I set memory_mask = None.

I'm not sure if this is correct to set the memory mask to None. If you take a look at generate_square_subsequent_mask function, you will see how to generate attention mask. It should be very straightforward..

[xdwang0726]

Thank you for your reply!
I used generate_square_mask to obtain the tgt_mask, but for memory_mask, I have some problem with the dimensions as generate_square_mask can only generate square masks which is not fit the memory_mask's dimension. So I set memory_mask = None.

I'm not sure if this is correct to set the memory mask to None. If you take a look at generate_square_subsequent_mask function, you will see how to generate attention mask. It should be very straightforward..

Thank you for your reply! As from the official documention, arg memory_mask is optional, I assumed that it can be set as None. I will try to generate the memory_mask and see whether it solves the problem. Thanks again for your help!

[zhangguanheng66]

If you set memory_mask to None, then why not just TransformerEncoder model?

[xdwang0726]

If you set memory_mask to None, then why not just TransformerEncoder model?

I went through the detailed implementation and found it seems that masks are MUSTs to nn.TransformerDecoder (however, in the documentation, it shows that masks are optional). For my understaing, there are two types of masks: attn_mask and padding_mask. The attn_mask is matrix with value '-inf's and 0s where -inf corresponed to value True and 0 is to False. And 'padding_mask' contains 1s and 0s to represent the paddings. However, in the documentation, it says

key_padding_mask: if provided, specified padding elements in the key will
be ignored by the attention. This is an binary mask. When the value is True,
the corresponding value on the attention layer will be filled with -inf.
attn_mask: mask that prevents attention to certain positions. This is an additive mask
(i.e. the values will be added to the attention layer).
https://pytorch.org/docs/stable/_modules/torch/nn/functional.html

I appreciate if you can answer this question. Thank you!

[zhangguanheng66]

If you set memory_mask to None, then why not just TransformerEncoder model?

I went through the detailed implementation and found it seems that masks are MUSTs to nn.TransformerDecoder (however, in the documentation, it shows that masks are optional). For my understaing, there are two types of masks: attn_mask and padding_mask. The attn_mask is matrix with value '-inf's and 0s where -inf corresponed to value True and 0 is to False. And 'padding_mask' contains 1s and 0s to represent the paddings. However, in the documentation, it says

key_padding_mask: if provided, specified padding elements in the key will
be ignored by the attention. This is an binary mask. When the value is True,
the corresponding value on the attention layer will be filled with -inf.
attn_mask: mask that prevents attention to certain positions. This is an additive mask
(i.e. the values will be added to the attention layer).
https://pytorch.org/docs/stable/_modules/torch/nn/functional.html

I appreciate if you can answer this question. Thank you!

According to the paper Attention is All You Need, mask is not a MUST. When it comes to WLM, we have to use a mask to avoid the model "copy" the word after in the sequence, IMO.

[hadaev8]

Same here, trying to make encoder-decoder model and cant find tutorial.

[zhangguanheng66]

Which kind of problem are you working on?

[hadaev8]

I have a seq2seq model for translating grapheme tp phoneme.
Here it is
https://colab.research.google.com/drive/1g4ZFCGegOmD-xXL-Ggu7K5LVoJeXYJ75

Loss immediately drops to zero and the model generates the same token at interference.
From things I read on google, it looks like something with a decoder mask and model just take the future token from the target input.
Still, I passed a mask and it everything should work.

[zhangguanheng66]

I think you should mask the inputs of both encoder/decoder. Based on what you observe, the masks were definitely not set up correctly.