I’m using ps moves. What I have said works for controllers as well.
Yaw offset in first section would rotate the controllers back to the front.
The positional drift over time happens to the entire setup: hmd and controllers together. So it’s not that noticeable. Unless it’s in beat saber or pirate trainer since game doesn’t recenter.