Xbox One (no matter the kind) had been plagued with issues for all sorts of games. Unfortunately, I am not sure how you can fix this. I had this issue too and it frustrated me so much I ended up moving over to PC to play the game.
There are rumors we are getting an updated version for newer consoles so fingers crossed for that, assuming you have a Series X/S.