Commit 68e0458
authored
Fix attention mask to use float_lowest instead of -inf and add NaN-safe softmax handling (#2654)
#2561
- Use lowest representable float value instead of -inf for attention
masks.
- Add NaN-safe handling and a unit test for softmax with all masked
positions.
Please let me know if my approach or fix needs any improvements . I’m
open to feedback and happy to make changes based on suggestions.
Thankyou !1 parent a9cb429 commit 68e0458
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2073 | 2073 | | |
2074 | 2074 | | |
2075 | 2075 | | |
2076 | | - | |
| 2076 | + | |
2077 | 2077 | | |
2078 | 2078 | | |
2079 | 2079 | | |
| |||
0 commit comments