News for hackers. Stuff that matters.

A comment on Meta is inviting researchers to pick apart the flaws in its version of GPT-3. Parent comment

Re:Meta is inviting researchers to pick apart the flaws in i... by axg11 on Monday June 27, 2022 @04:51PM (#31900621)
It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy.

Some interesting takeaways:

- Meta aren't using any software for scientific logbooks, just prepending a document

- So many hardware/cluster issues.

- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)

- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project

[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by sanxiyn on Monday June 27, 2022 @07:44PM (#31902397)
If you are interested in hot-swapping, I highly recommend OpenAI Five paper: https://arxiv.org/abs/1912.06680. It includes 10 months log of how they adapted the model to releases of Dota 2 version 7.19, 7.20, and 7.21.

I agree hot-swapping is underreported, but there are some good existing reports. In fact, OpenAI Five paper today is probably more valuable for its details on hot-swapping than its details on main model which used LSTM and not transformer.

[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by hoseja on Tuesday June 28, 2022 @01:49AM (#31904530)
Weird they felt the need to update the version since they used a very limited subset of Dota anyway.
[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by domenicrosati on Monday June 27, 2022 @05:02PM (#31900762)
I wonder what software would be good for a logbook like this... I just use google docs for these kinds of things. Sure wandb and jupyter notebooks are good but they are not so good for notes and ideas and documentation
[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by Gigachad on Monday June 27, 2022 @07:57PM (#31902476)
Sometimes the generic solution is just the best. No one requires special training on Google Docs and it just comes with handy features like version control and live updates.
[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by EricLeer on Tuesday June 28, 2022 @02:33AM (#31904777)
Personally I use notion for this. Not dedicated software, but the extra options in formatting/structuring and linking can make a document like this a lot more readible.
[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by semi-extrinsic on Tuesday June 28, 2022 @01:58AM (#31904568)
We tried jupyter notebooks for this kind of thing once, and after the equivalent of ~50 pages it becomes unusably slow in my experience.
[ Parent | on HN ]
Re:Meta is inviting researchers to pick apart the flaws in i... by asadlionpk on Monday June 27, 2022 @07:24PM (#31902244)
LogSeq? I use it for this kinda work...
[ Parent | on HN ]