Sounds like the perfect fit for machine learning. Let people continue classifying the input text, store the input text and classification, keep doing this for a while and you get a large training set. Regularly run ML on it and compare results. Switch to ML when its results are good enough.