Apple Sillicon MPS NaN error

[blimjoe]

When using large model/medium model on Apple Sillicon with MPS enabled, it's truely accelerated with full almost full gpu usage. But it will crash in few mins, due to error:

MPS numerical instability detected: Expected parameter logits (Tensor of shape (1, 51865)) of distribution Categorical(logits: torch.Size([1, 51865])) to satisfy the constraint IndependentConstraint(Real(), 1), but found invalid values:
tensor([[nan, nan, nan, ..., nan, nan, nan]], device='mps:0'

It seems to be a Apple Sillicon problem? Because of the deep Transformer layer? Or cumulative errors caused by insufficient mixed-precision support in Apple Silicon GPU MPS?

[blimjoe]

But it works perfect on turbo model. WHY?