News for hackers. Stuff that matters.
Re:The open source learning curve for AI researchers by godelski on Wednesday July 19, 2023 @11:13PM (#36796502)
I have a habit of rewriting code because it is hard for me to understand without doing this and frankly, it isn't uncommon for the algorithms to be different. But the most common thing is that a small one-liner is a critical component to making the model work. Sometimes this is a one-liner in the paper, sometimes this is a one-liner in the paper that they built their work on, of the paper they built their work on, of the paper they built their work on.

People building on top of one another's code definitely helps make projects get up and running faster, but I do wonder if it makes research faster and/or better. If a secret sauce gets lost in the game of telephone, it can be hard to know how much is the secret sauce. Especially when comparing to works that don't use that line. Sometimes the secret ingredient gets lost and similarly the gains. It's also not uncommon for this secret ingredient to not be known, or acknowledged. Especially if it is "well known" (won't be for long).

Re:The open source learning curve for AI researchers by techwizrd on Thursday July 20, 2023 @02:10PM (#36804371)
There is so much "secret sauce" that we're unconsciously relying upon. I learned a lot of secret sauces by reading through various implementations, "logbooks" [0] by various researchers, and so on. It definitely makes AI/ML feel more like an art than a science, and it makes it very difficult to re-implement in another language (e.g., Julia) or framework.

0: Such as https://github.com/facebookresearch/metaseq/blob/main/projec...