avisoori1x
Re:A Simple Version of Grok 1.5/ GPT-4 Vision from scratch, in one PyT...
on Monday May 06, 2024 @01:43PM (#40277257)
Please note that this is borrows heavily from GIT/ Kosmos from Microsoft and LLaVA
Re:Simple Multimodal LLM from Scratch
on Tuesday April 23, 2024 @04:00PM (#40136590)
Updated link here: https://avisoori1x.github.io/2024/04/22/seemore-_Implement_a...
Hugging Face community blogs seems to have gone down.
Simple Multimodal LLM from Scratch
Re:Implementation of vision language model in a single file of PyTorch
on Monday April 22, 2024 @08:40AM (#40113791)
I implemented a vision language model consisting of an image encoder, a multimodal projection module and a decoder language model in pure PyTorch. Think of this as a simplified version of what you see in GPT-4 or Claude 3 in terms of vision capabilities demonstrated by a language model. The name ‘seemore’ is my way of paying homage to Andrej Karpathy’s ‘makemore’ because here I use a character level autoregressive language model much like in his basic transformer implementation. seemore.py in the repo has the single file with everything in it.
Implementation of vision language model in a single file of PyTorch
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Databricks
on Wednesday March 27, 2024 @03:03PM (#39843259)
This repo I created and the linked blog will help in understanding this: https://github.com/AviSoori1x/makeMoE
Re:Implementation of mixture of experts language model in a single fil...
on Saturday March 23, 2024 @02:13PM (#39801936)
This is awesome! Thanks for sharing. I'll definitely check this out.
Re:Implementation of mixture of experts language model in a single fil...
on Saturday March 23, 2024 @02:12PM (#39801932)
Quite honestly not in my experiments. I wanted to do some Bayesian hyperparameter optimization with some discretized options like noise/no-noise and n_expert/top_k but haven't been able to find the time or free time in one of our GPU clusters. I plan on using perplexity as this is not yet instruction fine tuned.
Re:Implementation of mixture of experts language model in a single fil...
on Friday March 22, 2024 @02:50PM (#39793594)
Oh nice. What's new here would be noisy top-k routing and expert capacity. It also seems to use the nanoGPT base from Andrej Karpathy. Mine is from January as well. Here's the original blog: https://huggingface.co/blog/AviSoori1x/makemoe-from-scratch
Re:Implementation of mixture of experts language model in a single fil...
on Friday March 22, 2024 @01:23PM (#39792870)
This is a good point. I'm yet to try it as I've kind of let this project sit for a couple of months and only getting back to it. I went with this because it's simpler but I'm not sure simpler is necessarily better in this case.
Re:Implementation of mixture of experts language model in a single fil...
on Friday March 22, 2024 @01:22PM (#39792853)
Thanks! So this is something I tried and qualitatively I didn't see a huge difference. I'd like to swap out my hand rolled modules with standard pytorch modules for self attention etc. and train it on the wikipedia English split. That's on my to-do list for sure.
Implementation of mixture of experts language model in a single file of PyTorch
Re:Implementation of mixture of experts language model in a single fil...
on Monday March 18, 2024 @07:57AM (#39742922)
A from scratch implementation of a sparse mixture of experts language model in a single file of PyTorch. This is inspired by and largely based on Andrej Karpathy's project 'makemore' and borrows a number of re-usable components from that implementation. Just like makemore, makeMoE is also an autoregressive character-level language model but uses the aforementioned sparse mixture of experts architecture. I added Expert Capacity to this implementation to make it more complete
Re:Show HN: Fast, turnkey static web site recipe (Terraform and Jekyll...
on Saturday January 23, 2021 @08:58PM (#25888230)
This seems very useful. Nice work as always!