Re:Llama 3.1 Omni Model
by killerstorm on Thursday September 19, 2024 @03:11AM
(#41589356)
Speech is inherently easier to represent as a sequence of tokens than a high-resolution image.
Best speech to text is already NN transformer based anyway, so in theory it's only better to use a combined model
Interesting, I've actually been using gpt4o extensively for OCR and didn't encounter any significant issues - could I ask you to please give an example of an image of (otherwise legible) text that it hallucinates on?
Re:Llama 3.1 Omni Model
by schrodinger on Thursday September 19, 2024 @11:50AM
(#41593130)
Same, it's perfect at OCR. Generating an image with text in it however… nope!
You can work around this by the way by sending the output through a checking stage.
So picture -> gpt4o -> out1, picture -> tesseract -> out2, out1,out2 -> llm.
Might work for sound too.