This is right sort of UI. I want to be able to highlight text and say what type of entity is selected to train a new model.

It needs to be more of a workflow based system though, so you can upload 10s of thousands of documents and tag them as quickly as is reasonable.

I think BRAT can do that: http://brat.nlplab.org/index.html

I've never used it for doing the annotation myself though.

There are a few other possible tools here: https://omictools.com/text-annotation-category

This looks like what I need, thanks!