News for hackers. Stuff that matters.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by alecco on Sunday September 01, 2024 @03:40AM (#41415044)
Using PyTorch is not "LLMs from the ground up".

It's a fine PyTorch tutorial but let's not pretend it's something low level.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by menzoic on Sunday September 01, 2024 @04:00AM (#41415117)
Is this a joke? Can’t tell. OpenAI uses PyTorch to build LLMs
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jnhl on Sunday September 01, 2024 @04:16AM (#41415173)
You could always go deeper and from some points of view, it's not "from the ground up" enough unless you build your own autograd and tensors from plain numpy arrays.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by 0cf8612b2e1e on Sunday September 01, 2024 @12:47PM (#41418309)
Numpy sounds like cheating on the backs of others. Going to need your own hand crafted linear algebra routines.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by TZubiri on Sunday September 01, 2024 @04:29AM (#41415232)
Source please?
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by leobg on Sunday September 01, 2024 @05:04AM (#41415371)
People think of the Karpathy tutorials which do indeed build LLMs from the ground up, starting with Python dictionaries.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by krmboya on Sunday September 01, 2024 @09:21AM (#41416801)
From scratch is relative. To a python programmer, from scratch may mean starting with dictionaries but a non-programmer will have to learn what python dicts are first.

To someone who already knows excel, from scratch with excel sheets instead of python may work with them.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by wredue on Sunday September 01, 2024 @12:26PM (#41418144)
For the record, if you do not know what a dict actually is, and how it works, it is impossible to use it effectively.

Although if your claim is then that most programmers do not care about being effective, that I would tend to agree with given the 64 gigs of ram my basic text editors need these days.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by carlmr on Sunday September 01, 2024 @04:32PM (#41420129)
>For the record, if you do not know what a dict actually is, and how it works, it is impossible to use it effectively.

While I agree it's good to know how your collections work. "Efficient key-value store" may be enough to use it effectively 80% of the time for somebody dabbling in Python.

Sadly I've met enough people that call themselves programmers that didn't even have such a surface level understanding of it.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by atoav on Sunday September 01, 2024 @05:45AM (#41415547)
No it is not. From scratch has a meaning. To me it means: in a way that letxs you undrrstand the important details, e.g. using a programming language without major dependencies.

Calling that from scratch is like saying "Just go to the store and tell them what you want" in a series called: "How to make sausage from scratch".

When I want to know how to do X from scratch I am not interested in "how to get X the fastest way possible", to be frank I am not even interested in "How to get X in the way others typically get it", what I am interested in is learning how to do all the stuff that is normally hidden away in dependencies or frameworks myself — or, you know, from scratch. And considering the comments here I am not alone in that reading.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by kenjackson on Sunday September 01, 2024 @12:49PM (#41418320)
Your definition doesn’t match mine. My definition is fuzzier. It is “building something using no more than the common tools of the trade”. The term “common” is very era dependent.

For example, building a web server from scratch - I’d probably assume the presence of a sockets library or at the very least networking card driver support. For logging and configuration I’d assume standard I/o support.

It probably comes down to what you think makes LLMs interesting as programs.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by atoav on Monday September 02, 2024 @07:15AM (#41424404)
It is okay to differ on this. Language is not an exact science. It is however always good to factor in expectations when you describe things.

E.g. when a title says it shows you how to do a thing in vanilla javascript from scratch bringing in jquery in the first step makes that tile a lie. If you bring in a hefty dependency on step 1 and run three imported function the vanilla javascript part might be fine, but the from scratch starts to do some heavy lifting.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by SirSegWit on Sunday September 01, 2024 @04:11AM (#41415154)
I'm still waiting for an assembly language model tutorial, but apparently there are no real engineers out there anymore, only torch script kiddies /s
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by sigmoid10 on Sunday September 01, 2024 @04:16AM (#41415172)
Pfft. Assembly. I'm waiting for the real low level tutorial based on quantum electrodynamics.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by oaw-bct-ar-bamf on Sunday September 01, 2024 @05:04AM (#41415370)
Automotive actually uses ML in plain c with some inline assembly sprinkled on top run run models in embedded devices.

It’s definitely out there and in productive use.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by mdp2021 on Sunday September 01, 2024 @12:10PM (#41418032)
> ML in plain c

Which engines in particular? I never found especially flexible ones.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by wredue on Sunday September 01, 2024 @12:29PM (#41418165)
Ironically, slippery slope argumentation is a favourite style of kids.

Unfortunately, your argument is a well known fallacy and carries no weight.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by bvrmn on Monday September 02, 2024 @07:08AM (#41424369)
It's a natural response for "No true Scotsman" in the parent.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by botverse on Sunday September 01, 2024 @04:21AM (#41415194)
#378
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by alecco on Sunday September 01, 2024 @07:51AM (#41416149)
I'll write a guide "no-code LLMs in CUDA".
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jb1991 on Sunday September 01, 2024 @04:24AM (#41415214)
Learn to play Bach: start with making your own piano.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by defrost on Sunday September 01, 2024 @04:33AM (#41415249)
Bach (Johann Sebastian .. there were many musical Bach's in the family) owned and wrote for harpsichords, lute-harpsichords, violin, viola, cellos, a viola da gamba, lute and spinet.

Never had a piano, not even a fortepiano .. though reportedly he played one once.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jb1991 on Sunday September 01, 2024 @08:10AM (#41416296)
Yes, I know, but that’s irrelevant. You can replace the word piano in my comment with harpsichord if it makes you happy.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by generic92034 on Sunday September 01, 2024 @06:59AM (#41415855)
He had to improvise on the Hammerklavier when visiting Frederick the Great in Potsdam. That (improvising for Frederick) is also the starting point for the later creation of https://en.wikipedia.org/wiki/The_Musical_Offering .
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by vixen99 on Sunday September 01, 2024 @07:39AM (#41416068)
We know what he meant.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jahdgOI on Sunday September 01, 2024 @05:01AM (#41415361)
Pianos are not proprietary in that they all have the same interface. This is like a web development tutorial in ColdFusion.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by maleldil on Sunday September 01, 2024 @09:37AM (#41416880)
Are you implying that PyTorch is proprietary?
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jb1991 on Sunday September 01, 2024 @11:43AM (#41417828)
We’re digressing to get way off the whole point of the comment, but to address your point, actually piano design has been an area of great innovation over the centuries, with different companies doing it in considerably different ways.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by atoav on Sunday September 01, 2024 @05:36AM (#41415513)
Wanted to say the same thing. As an educator who once gave a course on a similar topic for non-programmers you need to start way, way earlier.

E.g.

1. Programming basics

2. How to manipulate text using programs (reading, writing, tokenization, counting words, randomization, case conversion, ...)

3. How to extract statistical properties from texts (ngrams, etc, ...)

4. How to generate crude text using markov chains

5. Improving on markov chains and thinking about/trying out different topologies

Etc.

Sure markov chains are not exactly LLMS, but they are a good starting point to byild a intuition how programs can extract statistical properties from text and generate new text based on that. Also it gives you a feeling how programes can work on text.

If you start directly with a framework there is some essential understanding missing.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by BaculumMeumEst on Sunday September 01, 2024 @06:11AM (#41415656)
I really like Sebastian's content but I do agree with you. I didn't get into deep learning until starting with Karpathy's series, which starts by creating an autograd engine from scratch. Before that I tried learning with fast.ai, which dives immediately into building networks with Pytorch, but I noped out of there quickly. It felt about as fun as learning Java in high school. I need to understand what I'm working with!
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by krmboya on Sunday September 01, 2024 @09:14AM (#41416758)
Maybe it's just different learning styles. Some people, me included, like to start getting some immediate real world results to keep it relevant and form some kind of intuition, then start peeling back the layers to understand the underlying principles. With fastAI you are already doing this by the 3rd lecture.

Like driving a car, you don't need to understand what's under the hood you start driving, but eventually understanding it makes you a better driver.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by BaculumMeumEst on Sunday September 01, 2024 @11:10AM (#41417594)
For sure! In both cases I imagine it is a conscious choice where the teachers thought about the trade-offs of each option. Both have their merits. Whenever you write learning material you have to decide where to draw the line of how far you want to break down the subject matter. You have to think quite hard about exactly who you are writing for. It's really hard to do!
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jph00 on Sunday September 01, 2024 @01:39PM (#41418716)
You seem to be implying that the top-down approach is a trade off that involves not breaking down the subject matter into as lower level details. I think the opposite is true - when you go top down you can keep teaching lower and lower layers all the way down to physics if you like!
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by jph00 on Sunday September 01, 2024 @01:36PM (#41418685)
fast.ai also does autograd from scratch - and goes further than Karpathy since it even does matrix multiplication from scratch.

But it doesn’t start there. It uses top-down pedagogy, instead of bottom up.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by BaculumMeumEst on Sunday September 01, 2024 @01:53PM (#41418817)
Oh that’s interesting to know! I guess I gel better with bottom up. As soon as I start seeing API functions I don’t understand I immediately want to know how they work!
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by delano on Sunday September 01, 2024 @07:59AM (#41416209)
If you want to make an apple pie from scratch, first you have to invent the universe.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by CamperBob2 on Sunday September 01, 2024 @10:56AM (#41417466)
After watching the Karpathy videos on the subject, of course.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by _giorgio_ on Sunday September 01, 2024 @08:41AM (#41416527)
Your comment is one of the most pompous that I've ever read.

NVDIA value lies only in pytorch and cuda optimizations with respect with pure c implementation, so saying that you need go lower level than cuda or pytorch means simply reinventing Nvidia. Good luck with that

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by alecco on Sunday September 01, 2024 @09:06AM (#41416709)
1. I only said the meaning of the title is wrong, and I praised the content

2. I didn't say CUDA wouldn't be ground up or low level (please re-read) (I say in another comment about a no-code guide with CUDA, but it's obviously a joke)

3. And finally, I think your comment comes out as holier than thou and finger pointing and making a huge deal out of a minor semantic observation.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by _giorgio_ on Thursday September 12, 2024 @05:04AM (#41518927)
There's always a lower level, until there's not.

Pytorch is low level enough to understand and interpret each and every passage. In pytorch, you can use builtin transformers, or code them yourself down to the "lowest" level in which there's still a theoretical meaning. So pytorch is just a tool and your comment was just pompous and empty.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by nerdponx on Sunday September 01, 2024 @08:54AM (#41416623)
Low level by what standards? Is writing an IRC client in Python using only the socket API also not "from scratch"?
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by badsectoracula on Sunday September 01, 2024 @09:16AM (#41416770)
Considering i seem to be the minority here based on all the other responses the message you replied to, the answer i'd give is "by mine, i guess".

At least when i saw the "Building LLMs from the Ground Up" what i expected was someone to open vim, emacs or their favorite text editor and start writing some C code (or something around that level) to implement, well, everything from the "ground" (the operating system's user space which in most OSes is around the overall level of C) and "up".

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by nerdponx on Sunday September 01, 2024 @10:01AM (#41417048)
The problem with this line of thinking is that 1) it's all relative anyway, and 2) The notion of "ground" is completely different depending on which perspective you have.

To a statistician or a practitioner approaching machine learning from a mathematical perspective, the computational details are a distraction.

Yes, these models would not be possible without automatic differentiation and massively parallel computing. But there is a lot of rich detail to consider in building up the model from first mathematical principles, motivating design choices with prior art from natural language processing, various topics related to how input data is represented and loss is evaluated, data processing considerations, putting things into context of machine, learning more broadly, etc. You could fill half a book chapter with that kind of content (and people do), without ever talking about computational details beyond a passing mention.

In my personal opinion, fussing over manual memory management is far afield from anything useful unless you want to actually work on hardware or core library implementations like Pytorch. Nobody else in industry is doing that.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by wredue on Sunday September 01, 2024 @12:24PM (#41418129)
Gluing together premade components is not “from the ground up” by most people’s definition.

People are looking at the ground up for a clear picture of what the thing is actually doing, so masking the important part of what is actually happening, then calling it “ground up” is disingenuous.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by nerdponx on Sunday September 01, 2024 @12:37PM (#41418234)
Yes, but "what the thing is actually doing" is different depending on what your perspective is on what "the thing" and what "actually" consists of.

If you are interested in how the model works conceptually, how training works, how it represents text semantically, etc., then I maintain that computational details are an irrelevant distraction, not an essential foundation.

How about another analogy? Is SICP not a good foundation for learning about language design because it uses Scheme and not assembly or C?

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by badsectoracula on Sunday September 01, 2024 @08:41PM (#41421549)
> The problem with this line of thinking is that 1) it's all relative anyway, and 2) The notion of "ground" is completely different depending on which perspective you have.

But if all is relative and depends on your PoV that implies that there isn't actually a problem here, right? :-P

I don't think there is anything wrong with "building up the model from first mathematical principles" as you wrote, it just wasn't what i personally had in mind with the "from the ground up" part.

And FWIW i'm not that stuck up on the "vim and C" aspect, i used those as an example that i expected most would understand and leave little room for misinterpretation in what you'd have to work with (i.e. very very little) and have to implement yourself (pretty much everything) - personally i'd consider it from "the ground up" even if it was in C#, D, Java, JavaScript or even Python, as long as the implementation was done in a way that didn't rely on 3rd party libraries so that whatever is implemented in, say, Java could also be implementable in C#, D, JavaScript or Python with just whatever is available out of the box in those languages or even C, if one doesn't mind writing the extra bookkeeping functionality themselves.

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by nerdponx on Tuesday September 03, 2024 @07:27AM (#41433767)
Right, but again I think the emphasis on avoiding 3rd party libraries isn't really relevant to machine learning. The "from scratch" here is avoiding 3rd party implementations of the transformer model, building up from the math on paper and then letting the AD/computation framework do its thing.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by badsectoracula on Wednesday September 04, 2024 @06:24AM (#41443969)
One does not exclude the other though. "Avoiding 3rd party implementations of the transformer model" is a subset of "avoiding 3rd party libraries". "From scratch" is, as seen, vague enough for different people to interpret it in different ways. Despite being the minority in this thread, i do not think my interpretation is any less valid - especially since some people have already done such "from scratch" (i.e. in C or C++ with no 3rd party dependencies) implementations already.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by bvrmn on Monday September 02, 2024 @07:13AM (#41424398)
In LLMs context pytorch.nn is low level. It's a crucial thing in education to not operate on too many abstraction levels.
Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by cruffle_duffle on Monday September 02, 2024 @12:32PM (#41426777)
Not just education but real life engineering too.