Make sure you're using hardware encoding if you can. Using the CPU encoder can be higher quality, but it can't beat the dedicated hardware on your GPU for encoding in terms of performance.

Even a first-gen Raspberry Pi 1 is able to encode 1080p30fps video with its hardware encoder.