This seems to have fixed it for me! Not that I could actually live with this fix, but now I could report it to them or something. When I turned it to default scaling it was snappy again. And when I move the window over to one of my 4K monitors it is also fast, so it's something unique to the laptop monitor with scaling.