Isn't OpenAI distributing content in its apps?

This is an extremely dangerous precedent that I think you are purposefully trying to put forward.

It's a horrendously bad idea especially for startups to make it apps' faults for how users use their platform. It's only in the benefit of entrenched tech companies to make this precedent.

No, whoever operates the LLM service is liable for unauthorized modification, reproduction and distribution of copyrighted work to users.

Ok, so is it ok if I run the whole thing on my own hardware, and never distribute?

If not, how does that differ from me making an unauthorized pencil drawing of Mario?

It's no different than pirating. No one will care unless you start sharing or try to start a business from it.