I do something like this by using old rule-based bot tech as the first layer you hit, and if it hits any situation it does not understand deterministically, it shells out for help to an external "smarter" agent.

That way you can do old school intent matching and rule-based bots that work super well, and only shell out for deeper work.