News for hackers. Stuff that matters.
Re:Deep Learning Systems (CMU Course) by smokel on Saturday August 12, 2023 @04:41PM (#37104065)
I really like the sneers about some terrible naming, e.g. in the slide on "The self-attention operation":

> “keys”, “queries”, “values”, in one of the least-meaningful semantic designations we have in deep learning

And in the context of LSTMs:

> throwing in some other names, like “forget gate”, “input gate”, “output gate” for good measure

This makes me feel more confident about actually understanding these topics. Before, I was totally misled by the awkward terminology.

Re:Deep Learning Systems (CMU Course) by sva_ on Saturday August 12, 2023 @05:02PM (#37104197)
It gets even worse when you see how the ML community likes to bastardize terms from neuroscience.

But I think it is good to have memorable names that one can use to talk about the concepts verbally.

Re:Deep Learning Systems (CMU Course) by jeron on Saturday August 12, 2023 @06:54PM (#37104965)
simply operator overloading
Re:Deep Learning Systems (CMU Course) by p1esk on Saturday August 12, 2023 @10:10PM (#37105995)
The names make sense to me once I understood what they represent. How else would you like to call them?
Re:Deep Learning Systems (CMU Course) by smokel on Sunday August 13, 2023 @02:59AM (#37107350)
I understand the terminology in the context of the original papers, but to me the metaphors don't seem to generalize well, or at least not in the suggested direction.

This is probably just an unfortunate situation, due to progressive understanding. Pointing this out in the slides gave me a sense of relief.

Personally, I wouldn't put names to every minor part of an algorithm or formula that was discovered to work empirically. But then again, I haven't discovered anything, and the authors of the respective papers certainly deserve some credit for their inventions!

Re:Deep Learning Systems (CMU Course) by visarga on Sunday August 13, 2023 @12:53AM (#37106810)
> “forget gate”, “input gate”, “output gate”

These are legit:

cell_h = cell_(h-1) * forget_gate + tanh(linear(input_h)) * input_gate

out_h = cell_h * output_gate

see? forget_gate masks input by multiplying with numbers in [0, 1], input_gate controls the external input, output_gate controls the output of course

Re:Deep Learning Systems (CMU Course) by 0cf8612b2e1e on Monday August 14, 2023 @12:54PM (#37123332)
I also enjoy how ML likes to twist statistical nomenclature enough to be irritating.