PaliGemma: Open-Source Multimodal Model by Google

Re:PaliGemma: Open-Source Multimodal Model by Google by sigmoid10 on Wednesday May 15, 2024 @04:59PM (#40372289)

You could have also just looked at the documentation linked in the original post...

[ Parent | on HN ]

Re:PaliGemma: Open-Source Multimodal Model by Google by simonw on Wednesday May 15, 2024 @05:11PM (#40372405)

Have you seen any documents that help explain how to interpret those tokens?

[ Parent | on HN ]

Re:PaliGemma: Open-Source Multimodal Model by Google by zerojames on Wednesday May 15, 2024 @05:23PM (#40372533)

(I work at Roboflow)

We're actively working on this!

Our ML team has noted that the segmentation masks in particular from PaLiGemma are a bit tedious and unintuitive to decode. We should be pushing out (more!) open source software that uses this model in the coming days.

Look forward to an easy way to fine-tune PaLiGemma and broader support for its task types in `inference`, the package used in the blog post.

[ Parent | on HN ]

Re:PaliGemma: Open-Source Multimodal Model by Google by llama_person on Wednesday May 15, 2024 @05:50PM (#40372843)

https://huggingface.co/spaces/google/paligemma/blob/main/pal...

the blog post details it but essentially to convert from PaliGemma tokens to bbox:

y0 / 1024 * h

x0 / 1024 * w

y1 / 1024 * h

x1 / 1024 * w

have not played with segmentation yet.

[ Parent | on HN ]

Re:PaliGemma: Open-Source Multimodal Model by Google by sigmoid10 on Saturday May 18, 2024 @04:00AM (#40397227)

Sure. Click on the github link in the post and look at the readme. It's all there. There is even a linked paper.

[ Parent | on HN ]