News for hackers. Stuff that matters.

A comment on In Defense of Pure 16-Bit Floating-Point Neural Networks. Parent comment

Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by imtringued on Tuesday May 23, 2023 @12:47PM (#36046964)
From what I can tell the architecture is more important anyway and having smaller but more parameters gives the model more chances to figure out the optimal architecture on its own.
[ Parent | on HN ]
Re:In Defense of Pure 16-Bit Floating-Point Neural Networks by tysam_and on Tuesday May 23, 2023 @02:23PM (#36048634)
My best understanding is that architecture is predetermined, which determines the number of parameters up front?

I do think that, however, having shallower bit depths over time will require some slightly deeper networks to compensate, as a result. Sorta makes sense when you think about it a bit. :) <3 :DDDD :)

[ Parent | on HN ]