What I was getting at is why Kinect messes up when it overlaps? For now, whenever they overlap(hand/hand, hand/body), they stick together. In the scenarios described above, you actually have enough data to separate them.
Similarly, if you can solve this, you can have 360 coverage (when the hands/arms are behind the body; head facing away from sensor) for hand positions.
And there’s the phone orientation problem left…
It takes away some need of having a joystick since the current mapping scheme works somewhat: press and orient to select.