Open Flamingo – open framework to train multimodal LLMs

Re:Open Flamingo – open framework to train multimodal LLMs by yeldarb on Tuesday March 28, 2023 @05:54PM (#35348500)

I always like to try these zero-shot models on things outside of the "normal" COCO classes. Here are some chess board queries:

Counting: https://imgur.com/KTuQ1Bv

Parse the chess board: https://imgur.com/2zYFK1P

(Result): https://imgur.com/Ei4MAl7

Few-Shot Object Detection (Pascal VOC): https://imgur.com/gZkDMn8

Few-Shot Object Detection (simplified): https://imgur.com/Hk8QGMd

Not quite there yet. I've been more impressed with the other new zero-shot multimodal models like Grounding DINO and Azure Dense Captioning. Really looking forward to putting multimodal GPT-4 through its paces as well.

[ Parent | on HN ]

Re:Open Flamingo – open framework to train multimodal LLMs by rikimaru0345 on Tuesday March 28, 2023 @10:08PM (#35351248)

> Parse the chess board:

Could it be that the actual issue has to do with it having trouble with small tokens (letters, numbers)?

Does it give a different result if you ask it to answer in a format like this?

> Please name what kind of piece is on each square of this board > A1: white rook > A2: white pawn > A3: empty > A4: empty > ...

Prompting can be so unintuitive sometimes. Maybe it just has an issue with the output representation or something...

[ Parent | on HN ]