I have noticed that the drifting happens due to two things.
First, the head movement recorded is transformed in movement in the "tape", but it looks more of a soft fluctuation.
Second, it appears that the VR boundaries are turned on when playing the recording, even if they are not. And if you move your head to the sides or to the front during the playback, it pushes the aircraft in the direction of your movement. It does not matter if the recording happens in Normal, Expert or Custom mode.
As to minimize this effect, I try to not move my head during recording to prevent the first reason but the second still happens.