What the title says.I am trying to figure out how much people expect to spend when training their own LLM on their own data from scratch. I’m also wondering if there are techniques you can use to get that cost down (data/model parallelism. Pipelines etc.) and what libraries implement these.Thanks!
Arxiv + google + papers with code is how I do it. As another user pointed out, focus on solving your problem. By doing this, you'll search and then naturally find relevant papers. After a while, a lot of papers become super easy to read because they're conceptually easy to grasp. Once you hit this point it becomes extremely easy to skim the content that you're searching for. Once I identify something interesting in my problem space or adjacent then getting through it is the hard part.
TLDR: Focus on a problem domain and search within it. Don't just shotgun it. If its relevant enough to you it will show up. Use filters like HN and whatnot.
I posted previously about this, but I recently build a model that seems to be outperforming most other architectures I've seen. Wondering how to test it's full context window without spending massive amounts of money training.
So, my friend and I built a custom model that is performing 2-3X as good as a comparable decoder only model (~same params, exact same input data etc.) and it's doing this on CodeXGlue.I am currently trying to figure out what the best method would be to a) scale it, b) test it on other benchmarks, and c) how to really determine max context length. This last one is really the point of interest....
TLDR: big companies make networking easy because of crud. Pretty much all software companies are just crud apps (maybe with some sockets stuff if they’re cool)
Hello, it’s my first time posting here. I figured this would be a good place to discuss my current concerns about modern infrastructure management abstractions. I am hoping to have a fruitful discuss about how to best tackle multi scale infra management (i.e. side project -> FAANG) in a way that is easy for devs to manage. I basically have 3 MAJOR problems with most infrastructure solutions....