I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)
It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.
That seems arguably pretty cheap compared to the human alternatives.
I do get super annoyed at how specific you've got to be with these tools. They are like almost like malicious compliance geniuses lol.
They're like a monkey's paw or evil genie I think.
You'll get exactly what you ask for, but lots of weirdness you didn't expect.
To be fair to the model makers, deciding what to do with an under-specified prompt is hard, particularly in an RL environment where you can't just ask the user.
For stuff I want to run for a while, I'm defaulting to telling it to ask me the questions upfront to reduce the incidence of these kinds of problems.
To get the best performance you really need to present your problem as an iterative verification loop.
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.