Rotation gizmo instability

[Glorou]

Hi Cedric,

Wanted to get your input on this bug I've been having, its somewhat hard to describe but you can see it clearly in this video, the gizmo is unstable in some scenarios where the mouse moves closer or further from it, I think this might be related to #250 in some way, I just wanted to hear your thoughts to begin doing bug fixing and hopefully I can make a PR

ffxiv_dx11_d5ir1xOUh3.mp4

[CedricGuillemet]

Yes, when plane a not parallel enough, it gets hard to get precise control. 1 solution is to not allow the control and hide the specific gizmo rotation circle. Ideally, it should behave like Maya or Blender.

(Nice rendering BTW!)

[Glorou]

So I realized the demo I showed was a bit limited, this happens even if its roughly parallel as you can see in the attached, I'm just contributing but I'll pass it along!

ffxiv_dx11_5c6VTGQ89Y.mp4

ffxiv_dx11_eTU3HRY6CV.mp4

[Glorou]

Hi again Cedric,

I've attached a video of some more rotation gizmo oddities, this one happens when somthing external to the gizmo acts on the matrix being rotated while you are manipulating it (e.x. IK calcs on bones), would just be nice to have the angle indicator stay in line with the axis circle, wanted to hear your thought on what might be wrong, I'm looking in to the rotation gizmo code now.

ffxiv_dx11_YQpNbQRy1z.mp4

[CedricGuillemet]

previous message error is linked to horizon I think. When the raycasting can't pick the plane (shooting the sky). There is some kind of flip I'm sure why it's not consistent.
Last issue will need to be fixed on your side. Matrix is used for rendering. For the plane being edited, a cached value is used (that's why it's stable). But for the rest of the gizmo the matrix is used to show what's happening on other planes. Is there a way for you to decompose/recompose the matrix?