torch.float32 sigmas when using torch.float16 training

[Binyr]

Hi,

This line will sample a torch.float32 sigmas.(

SVD_Xtend/train_svd.py

Line 964 in 609fbf9

sigmas = rand_log_normal(shape=[bsz,], loc=0.7, scale=1.6).to(latents.device)

). As a result, inp_noisy_latents is also converted to torch.float32 (

SVD_Xtend/train_svd.py

Line 972 in 609fbf9

inp_noisy_latents = noisy_latents / ((sigmas**2 + 1) ** 0.5)

). The same problem happens to (

SVD_Xtend/train_svd.py

Line 969 in 609fbf9

timesteps = torch.Tensor(

).

Is this torch.float32 type tensor necesaary?

Thank you,
Yanrui