News for hackers. Stuff that matters.
Re:Notes on training BERT from scratch on an 8GB consumer GPU by 2-718-281-828 on Friday June 02, 2023 @03:33AM (#36162202)
i've always been curious about looking at the statistics of comments/points of hn posts. so this is up 10h, has 57 points and not a single comment so far. fascinating.
Re:Notes on training BERT from scratch on an 8GB consumer GPU by dwrodri on Friday June 02, 2023 @03:58AM (#36162390)
For what it's worth, I think the lack of comments is due to the highly academic nature of the experiment displayed in TFA.

LLMs are all of the rage now, and the pre-training techniques used for BERT ended up laying the foundation for what became GPT pre-training runs. But I would assume even among HN users, there really are only a small handful of people interacting with this side of the NLP craze on a regular basis.

Compare that with the amount of users who probably have at least dabbled in using a programming language, and suddenly you see why the opinions fly everywhere.

A lot of the practical aspects of training neural nets are still more "I'm doing it this way because Radford et al did it" or "when you picture the problem in three dimensions, this generally seems like it should work" instead of "This was demonstrated to be the best way and I can show you it's the best way from first principles." because quite frankly that's the best MO we have when dealing with the statistical principles guiding billions of matrix multiplies for trillions of different token sequences.

Re:Notes on training BERT from scratch on an 8GB consumer GPU by dx034 on Friday June 02, 2023 @05:14AM (#36162921)
I'm interested in the topic but have no understanding in depth. I like upvoting stories like these to revisit them after a day when there are hopefully some interesting comments and/or clarifications.

I really like the high signal to noise ratio in HN comments.

Re:Notes on training BERT from scratch on an 8GB consumer GPU by gpderetta on Friday June 02, 2023 @07:48AM (#36163887)
So do I. Often I upvote interesting stories with little or no comments if I would like to read a discussion on the topic but I don't have anything worth to contributing personally.
Re:Notes on training BERT from scratch on an 8GB consumer GPU by Zemtomo on Friday June 02, 2023 @06:47AM (#36163490)
US is still sleeping btw.