Implementation Status and planned TODOs

[Rayhane-mamah]

this umbrella issue tracks my current progress and discuss priority of planned TODOs. It has been closed since all objectives are hit.

Goal

• achieve a high quality human-like text to speech synthesizer based on DeepMind's paper
• provide a pre-trained Tacotron-2 model (Training.. checking this still)

Model

Feature Prediction Model (Done)

• Convolutional-RNN encoder block
• Autoregressive decoder
• Location Sensitive Attention (+ smoothing option)
• Dynamic stop token prediction
• LSTM + Zoneout
• reduction factor (not used in the T2 paper)

Wavenet vocoder conditioned on Mel-Spectrogram (Done)

• 1D dilated convolution
• Local conditioning
• Global conditioning
• Upsampling network (by transposed convolutions)
• Mixture of logistic distributions
• Gaussian distribution for waveforms modeling
• Exponential Moving Average (train + synthesis)

Scripts

• Feature prediction model: training
• Feature prediction model: natural synthesis
• Feature prediction model: ground-truth aligned synthesis
• Wavenet vocoder model: training (ground truth Mel-Spectrograms)
• Wavenet vocoder model: training (ground truth aligned Mel-Spectrograms)
• Wavenet vocoder model: waveforms synthesis
• Global model: synthesis (from text to waveforms)

Extra (optional):

• Griffin-Lim (as an alternative vocoder)
• Reduction factor (speed up training, reduce model complexity + better alignment)
• Curriculum-Learning for RNN Natural synthesis. paper
• Post processing network for Linear Spectrogram mapping
• Wavenet with Gaussian distribution (reference)

Notes:

All models in this repository will be implemented in Tensorflow on a first stage, so in case you want to use a Wavenet vocoder implemented in Pytorch you can refer to this repository that shows very promising results.

[Rayhane-mamah]

Just putting some notes about the last commit (7e67d8b) to explain the motivation behind such major changes and to verify with the rest of us that I didn't make any silly mistakes ( as usual.. )

This commit mainly had 3 goals: (other changes are minor)

• Clean the code: added some comments and changed the code architecture to use tensorflow's attention wrapper in the objective of reducing the number of files. Even though I tried getting rid of the "custom_decoder" and the custom "dynamic_decode" I'm currently using, after diving deep into tensorflow's implementation, I found that it was impossible to adapt my dynamic <stop_token> prediction to use tensorflow's ready to use "BasicDecoder" and "dynamic_decode" with my custom helpers.
• Correct the Attention: Even though they called it "Location sensitive attention" in the paper, they didn't mean the "location based attention" we know but instead, they were mentioning the "hybrid" attention. For this hypothesis i'm relying on this part of the paper "We use the location sensitive attention from [21], which extends the additive attention mechanism [22] to use cumulative attention weights from previous decoder time steps as an additional feature." which supposes they took the original bahdanau attention and added location features to it."
• Added "map" (log) file at synthesis: Mainly, this will map each input sequence, to the corresponding real Mel-Spectrograms and generated ones.

I also want to bring attention to these few points (in case someone want to argue them):

• I impute finished sequences at decoding time to ensure model doesn't have to learn to predict paddings (which will probably result in extra noise in generated waveforms later)
• If I'm not mistaking, the paper writers used the projection to a scalar+sigmoid to explicitly predict a "<stop_token>" probability since our feature prediction model isn't performing some classification task where he can choose to output a real <stop_token>. I like to think of it as creating a small binary classifier that chooses when to stop decoding since vanilla decoder can't output a frame with full round zeros.
• I am only using 512 LSTM units for decoder in each layer as i supposed "The pre-net output and attention context vector are concatenated and passed through a stack of 2 uni-directional LSTM layers with 1024 units." means that the 1024 are distributed across the 2 layers.

[i-celeste-aurora]

Hi @Rayhane-mamah, using 7e67d8b I got an error that (in the end). You change he call -parameter name from previous_alignments to state in attention.py:108

Was that on purpose? AttentionWrapper from TF requires the parameter to be named previous_alignments. (Using TF 1.4)

Changing that back to previous_alignments results in other errors:

ValueError: Shapes must be equal rank, but are 2 and 1 for 'model/inference/decoder/while/BasicDecoderStep/decoder/output_projection_wrapper/output_projection_wrapper/concat_lstm_output_and_attention_wrapper/concat_lstm_output_and_attention_wrapper/multi_rnn_cell/cell_0/cell_0/concat_prenet_and_attention_wrapper/concat_prenet_and_attention_wrapper/attention_cell/MatMul' (op: 'BatchMatMul') with input shapes: [2,1,?,?], [?,?,512].

Any ideas?

[Rayhane-mamah]

Hi @imdatsolak, thanks for reaching out.

I encountered this problem on one of my machines, updating tensorflow to latest version solved the problem. (I changed the parameter to state according to latest tensorflow attention wrapper source code, I also want to point out that I am using TF 1.5 and confirm that attention wrapper works with "state" for this version and later).

Try updating tensorflow and keep me notified, I'll look into it if the problem persists.

[i-celeste-aurora]

@Rayhane-mamah, I tried with TF 1.5, which didn't work. Looking into TF 1.5, the parameter was still called previous_alignments. The parameter's name changed in TF 1.6 to state, so installed TF 1.6 and it works now. Thanks!

[danshirron]

Upgrading to TF1.6 (was 1.5) solved issue (TypeError: call() got an unexpected keyword argument 'previous_alignments') for me.

[Rayhane-mamah]

@imdatsolak, yes my bad. @danshirron is perfectly right. I checked that my version is 1.6 too (i don't remember updating it Oo)

[Rayhane-mamah]

Quick notes about the latest commit (7393fd5):

• Corrected parameters initialization which was causing gradients explosion in some cases (using a xavier initializer)
• Added gradients norm visualization
• Replaced the learning rate decay to start from step 0 (instead of 50000) and added a visualization of the learning rate
• Corrected typos in "hparams.py"
• Changed alignment plot directories and added real + predicted Mel-Spectrogram plots (each 100 training step)
• Added a small jupyter notebook where you can use griffin-lim to reconstruct phase and listen to the audio reconstructed from generated Mel-spectrograms (just to control the model learning state without paying much attention to audio quality as we will use wavenet as a vocoder)
• Started using a reduction factor (despite not being used in tacotron-2) as it speeds training process (faster computation) and allows for faster alignment learning. (current: r=5, feel free to change it).
• Corrected typos in preprocessing (Make sure to restart the preprocessing before training your next model)

Side notes:

• Alignment should appear at step 15k and audio becomes quite audible at 4~5k steps (using 32 batch size) but fully understandable around 8~10k steps.
• Mel-spectrograms seem very blurry at the beginning, and despite the loss not decreasing much (you may even feel it's constant after 1k steps) the model will still learn to improve speech quality so be patient.

If there are any problems, please feel free to report them, I'll get to it as fast as possible

[Rayhane-mamah]

Quick review of the latest changes (919c96a):

• Global code reorganization (for easier modifications and it's just cleaner now)
• Network Architecture review: Since there are some unclear points in the paper, I am doing my best to collect enough information from all related works, and trying to put them all together to get reasonable results. The current architecture is the closest I got to the described T2 (I think.. ^^')
• Pulled the <stop_token> prediction out of the decoder and got rid of the custom "dynamic_decode".
• Reduced the model size and added new targets (stop token targets are now prepared in the feeder)
• Adapted <stop_token> prediction to work properly with the reduction factor. (multiple <stop_token> predictions at each decoding step)
• Doubled the number of LSTM units in the decoder and number of neurones in the prenet. On the other hand, I removed the separate attention LSTM and started using the first decoder LSTM hidden state as a query for the attention.

Side Notes:

• Despite slightly reducing the memory usage of the model, impact of training speed are still not clear enough. Forward propagation got slightly faster and back propagation slightly slower. But the overall speed seems the same.

If anyone tries to train the model, please think about providing us with some feedback. (especially if the model needs improvement)

[ohleo]

Hi @Rayhane-mamah, thanks for sharing your work.

I cannot get a proper mel-spectrogram prediction and audible wave by Evaluation or Natural synthesis(No GTA) at step 50k.
All hparams are same with your code(with LJSpeech DB) and wave are generated by mel-prediction, mel_to_linear, Griffin-Lim reconstruction.
GTA synthesis generates audible results.

Is it works in your experiments?

I attached some Mel-spectrogram plot samples with following sentences.

1 : “In Dallas, one of the nine agents was assigned to assist in security measures at Love Field, and four had protective assignments at the Trade Mart."

Ground Truth

GTA

Natural(Eval)

2 : ”The remaining four had key responsibilities as members of the complement of the follow-up car in the motorcade."

Ground Truth

GTA

Natural(Eval)

3 : “Three of these agents occupied positions on the running boards of the car, and the fourth was seated in the car."

Ground Truth

GTA

Natural(Eval)

[Rayhane-mamah]

Hello @ohleo, thank you for trying our work and especially for sharing your results with us.

The problem you're reporting seems the same as the one @imdatsolak mentionned here.

There are two possible reasons I can think of right now:

• Your model after 50k steps still has an ugly allignment (hopefully this commit takes care of that). That's the most probable reason i think.
• I am unknowingly and indefinitely passing the first frame to the decoder in my code. I will triple check this today ( in case TacotestHelper is the cause).
• It can't be possibly doing a massive overfit on the first generated frame can it ? Oo the output looks the same for the three sentences!

The fact that GTA is working fine highly supposes the problem is in the helper.. I will report back to you later tonight.
If your setup is powerful enough, you could try to retrain the model using the latest commit or wait for me to test it myself a bit later this week.

In all cases, thanks a lot for your contribution, and hopefully we get around this issue soon.

[unwritten]

Hello, @Rayhane-mamah ,

do you get any further information running using the latest code?

[Rayhane-mamah]

Hello @unwritten, thanks for reaching out.
I believe you asked about GTA as well? I'm just gonna answer it anyway in case anyone gets the same question.

GTA stands for Ground Truth Aligned. Synthesizing audio using GTA is basically using teacher forcing to help the model predict Mel-spectrograms. If you aim to use the generated spectrograms to train some vocoder like Wavenet or else, then this is probably how you want to generate your spectrograms for now. It is important to note however that in a fully end-to-end test case, you won't be given the ground truth, so you will have to use the "natural" synthesis where the model will simply look at its last predicted frame to output the next one. (i.e: with no teacher forcing)

Until my last commit, the model wasn't able to use natural synthesis properly and I was mainly suspecting the attention mechanism because, well, how is the model supposed to generate correct frames if it doesn't attend to the input sequence correctly.. Which brings us to your question.

So after a long week end of debugging, it turned out that the attention mechanism is just fine, and that the problem might have been with some Tensorflow scopes or w/e.. (I'm not really quite sure what was the problem). Anyway, after going back through the entire architecture, trying some different preprocessing steps and replacing zoneout LSTMs with vanilla LSTMs, the problem seems to be solved (Now I am not entirely 100% sure as I have not yet trained the model too far, but things seem as they should be in early stages of training).

I will update the repository in a bit (right after doing some cleaning), and there will be several references to papers used that the implementation was based on. These papers will be in pdf format in the "papers" folder, like that it's easier to find if you want to have an in depth look of the model.

I will post some results (plots and griffin lim reconstructed audio) as soon as possible. Until then, if there is anything else I can assist you with, please let me know.

Notes:

• It is possible for now to use the griffin lim algorithm (using the provided notebook) to do a basic inversion of the mel spectrogram to waveform. The quality won't be as good as Wavenet's but it's mainly for test and debug purposes for now.
• Generated spectrograms using the "synthesize.py" will be stored under "output" folder. Depending on the synthesis mode you used, there will many possible sub-folders.
• I have not yet added the Wavenet vocoder to this repository as there are more important things at the moment like ensuring a good spectrograms generation. There are good Wavenet implementations out there that are conditioned of Mel-spectrograms like r9y9/wavenet.

[Rayhane-mamah]

Hello again @unwritten.

As promised I pushed the commit that contains the rectifications (c5e48a0).

Results, samples and pretrained model will be coming shortly.