I thought of a few ways you could improve the default WMR controller detection using the Kinect skeleton, but I’m not sure if they’d be possible for you to do. I’ll give them anyways in case you want to investigate them. Perhaps you could add them all and let the user choose the way they want it to act?
A few things to note regarding the controller tracking of the WMR:
– It has a limited range because of the headset tracking (Field of View)
– The controller rotations seem to be always accurate regardless of their position, they should always be used if possible.
– When a controller is out of range, it will completely freeze in place position-wise until it comes in range again. Rotation still works.
– I couldn’t find anywhere if you can check if the controller tracking was lost or not. I do notice the controllers start blinking when they are out of range, though. At least, in the Mixed Reality Portal, I can see the controller status goes from Ready to “Lost Tracking”.
Now, for the ideas/suggestions on how to improve the tracking, (Note, they are separate ideas, unrelated to each other). Since the controller rotation is always accurate regardless of its position, it should only override positions, not rotations. Anyway, here they go:
– Use the controller positions until goes out of range relative to the headset rotation (perhaps a calibration is needed to calculate the area where the controller goes out of range?) then Kinect hand position takes over.
– When controler stays in the same position for too long, d4vr assumes it lost tracking and lets the Kinect skeleton takes over.
– Another one could be to always use kinect skeleton but still reads the controller rotation. Try to use controller if the Kinect skeleton is too obfuscated
– (Probably one of the most practical) If the Kinect skeleton hand is too far from the controller hand, and the Kinect skeleton isn’t obfuscated, assume it has lost positional tracking and let the Kinect skeleton take over said hand. This method could be good in case a controller is obfuscated by the other (It has happened to me, at least).
– (Assuming you can read if the controller is correctly tracked or not) Let Kinect simply take over a hand while a controller has lost tracking.
Hope you find any of these ideas useful 🙂