tmabraham

Sophont Inc. | Founding Research Engineer / Scientist | Remote | Full-time

Sophont is a public-benefit corporation building open multimodal foundation models for medicine. Joining Sophont means you are paid to lead and contribute to high-impact medical AI research ultimately aimed to transform healthcare and life sciences.

We are looking for exceptional, high-agency ML research engineers who are passionate about using AI to build the future of healthcare. No previous background in medical AI is needed!

If you’re interested in working on highly impactful AI projects that could have the potential to transform and save many lives, Sophont is the company to work at.

## Roles

Founding Research Engineer, LLM Team - https://sophont.med/job_postings/llm Founding Research Scientist, Vision Team - https://sophont.med/job_postings/vision Founding Research Scientist, Miscellaneous - https://sophont.med/job_postings/misc

How to apply:

Email resume to [email protected]

I'm the co-founder of Sophont, happy to answer any questions!

it was already planned for open-sourcing, the leak did not affect the plans in any way

Nicolas Bourbaki is the name of a character in twenty one pilots' concept album

"He'll always try to stop me, that Nicolas Bourbaki He's got no friends close, but those who know him most know He goes by Nico"

It is indeed inspired by the actual pseudonym, not just a pun.

The paper has preliminary results for video as well

Because you can take advantage of pretrained CNNs and perform transfer learning, which is significantly more data-efficient than training from scratch, which is what you'd likely have to do with raw digital signals. This paper is not unique in this approach and many papers have obtained SOTA results by processing digital signals as images.

Yeah using data from a 7T MRI giving higher spatial resolution definitely helps!

The fMRI dataset includes signal from the whole brain but we only use the data from the visual cortex for this study.

Our model generates CLIP image embeddings from fMRI signals and those image embeddings can be used for retrieval (using cosine similarity for example) or passed into a pretrained diffusion model that takes in CLIP image embeddings and generates an image (it's a bit more complicated than that but that's the gist, read the blog post for more info).

So we are doing both reconstruction and retrieval.

The reconstruction achieves SOTA results. The retrieval demonstrates that the image embeddings contain fine-grained information, not just saying it's just a picture of a teddy bear and then the diffusion model just generates a random teddy bear picture.

I think the zebra example really highlights that. The image embedding generated matches the exact zebra image that was seen by the person. If the model only could say it's just a zebra picture, it wouldn't be able to do that. But the model is picking up on fine-grained info present in the fMRI signal.

The blog post has more information and the paper itself has even more information so please check it out! :)

Yes we've been looking into ControlNet as well, and I think there is one recent fMRI-to-image paper that also has tried ControlNet. Maybe we'll use ControlNet in MindEye v2 :)

We think it could be useful for clinical research and maybe even diagnostics. For example, you could imagine a person with depression(or other neurological disorders) may have a different perception of the same image than a healthy person. Now with the much higher fidelity that both more powerful MRI machines and better generative AI tools can provide, this may now be a very promising direction for future research.

I use GitHub Copilot very frequently for any programming I do.

I also use BingChat very often, I consider it to be a very underrated tool that people haven't fully explored. Of course, it's great for when you want your LLM queries augmented with search results. But BingChat can also view the current webpage, so for example you can pull up papers and ask BingChat questions about it. I've used it to help write abstracts for my papers. When I do this, BingChat literally searches up "how to write an abstract" which I found absolutely hilarious , it's like it's learning skills on the go as well. I didn't use the outputs directly, but it served as useful inspiration. I think BingChat uses a mix of models including GPT-4, all for free! Apparently more features are coming soon, including multimodal features.

Okay, but are you a diffusion model/generative AI researcher?

As a researcher in this field, I found the article fairly understandable and accessible. I suspect that was the target audience.

The domain of diffusion models have a lot of terminology and concepts that are not explained here, so maybe that's why it's harder to understand?