It strongly depends on what type of structure you can assume and how expensive sampling is. Dreamerv2, agent57 on Atari, dreamerv2 and the generalized agent model trained on 600 tasks by deepmind might be worth looking into for different approaches on pomdps, but you can do much better if you impose physics priors by e.g. using neural ODEs for the latent state modeling.

POMDP just means "observations are not state" and that you need to use a stateful policy to infer the state somehow, but without further assumptions it's difficult to answer this question