this is not something I officially tested but this is how I think it should work.
in such setup you should have only option head tracking enabled as Kinect – Skeleton.
then yo use controllers via psmoveservice and psmoveservicebridge.
you need somehow align controller position with head position when you start vr cause these are two different tracking systems.
the problem that you describe seems to be solely related to colour or pose calibration of psmoveservice. not really related to driver4vr.
i would recommend starting psmoveservice config tool. then preview controller position and it should have same behaviour as you see in vr.
when you see that, then ensure that each camera is tracking properly the led ball.
alternatively repeat calibratino mat steps. in case of problems with that I would search for help on psmoveservice forum.
that is my understanding of what happens, please let me know if that helps.