Have you tried the open source splash inference engine? I have the same setup as you and find it fast enough for most tasks.

https://inco.ai/blog/splash/

https://github.com/incoai/splash