News for hackers. Stuff that matters.
Re:Floating-Point Formats and Deep Learning by quietbritishjim on Tuesday July 28, 2020 @07:29AM (#23974589)
But even if you have extra precision around 1.0, assuming that weight is used for multiplication (of an output from the previous layer), surely you'll lose it as soon as you perform that multiplication? E.g. if the previous layer output is 3.5 and you multiply it by 1.00...0063, you'll get 3.500...02205, but you'll lose those trailing places on the output unless you have lots of precision around 3.5 too.

I tried to think about ways round that with traditional floating point formats and nonlinear functions (i.e. using exp to change the multiplication to addition) but it seems to come back to the same problem.

There are no comments on this comment yet.