Whatever the problem is, it probably isn't SDL. Here's a test project I worked on[0], and I'm using a garbage laptop. The sprites aren't that big but if you're just using a single texture it shouldn't matter, since SDL does sprite batching anyway.

Your theory might be right - the first thing I would look for was something allocating every frame.

You might ask the SDL Discourse forum and see what they think: https://discourse.libsdl.org/

[0]https://cdn.masto.host/krappmastohost/media_attachments/file...