Give Cython a try, you can get huge gains in memory/speed by carefully transitioning from Python to C with it. (Or use it to "write C in Cython"[0])

[0] https://explosion.ai/blog/writing-c-in-cython