I'm on Xbox Series X and I'm experiencing similar issues. The game is running smooth except for one thing: The PCI is laggy and jittery. Like it's running at a different framerate than everything else.
This issue was present in the Tech Test when I played on my Series X and it's really frustrating that it's made it's way to the final game. There were a number of posts about it in the Tech Test forum.