This sounds like an issue with something else you have installed in Emacs. I suggest running the profiler to see what is actually taking time. Org-drill itself on an otherwise vanilla config is fast, so it could be that you have hooks that are slow or something.