GPU memory is a key limit here and RL worsens the problem because it requires some eligibility trace or memory system. One option you could try would be to store past S,A,R,S' tuples on disc using something like DeepMind Reverb, and have a small batch size and simpler model. Or as mentioned in other comments, you can just use CPU and RAM which is often a lot higher capacity depending on your rig.