News for hackers. Stuff that matters.
Re:LLMs use a surprisingly simple mechanism to retrieve some... by vsnf on Thursday March 28, 2024 @11:26AM (#39852737)
> Linear functions, equations with only two variables and no exponents, capture the straightforward, straight-line relationship between two variables

Is this definition considering the output to be included in the set of variables? What a strange way to phrase it. Under this definition, I wonder what an equation with one variable is. Is a single constant an equation?

Re:LLMs use a surprisingly simple mechanism to retrieve some... by hansvm on Thursday March 28, 2024 @12:18PM (#39853554)
It's just a change in perspective. Consider a vertical line. To have an "output" variable you have to switch the ordinary `y=mx+b` formulation to `x=c`. The generalization `ax+by=c` accommodates any shifted line you can draw. Adding more variables increases the dimension of the space in consideration (`ax+by+cz=d` could potentially define a plane). Adding more equations potentially reduces the size of the space in consideration (e.g., if `x+y=1` then also knowing `2x+2y=2` wouldn't reduce the solution space, but `x-y=0` would, and would imply `x=y=1/2`, and further adding `x+2y=12` would imply a lack of solutions).

Mind you, the "two variable" statement in this news piece is a red-herring. The paper describes higher-dimension linear relationships, of the form `Mv=c` for some constant matrix `M`, some constant vector `c`, and some variable vector `v`.

On some level, the result isn't _that_ surprising. The paper only examines one layer (not the whole network), after the network has done a huge amount of embedding work. In that layer, they find that under half the time they're able to get over 60% of the way there with a linear approximation. Another interpretation is that the single layer does some linear work and shoves it through some nonlinear transformations, and more than half the time that nonlinearity does something very meaningful (and even in that under half the time where the linear approximation is "okay", the metrics are still bad).

I'm not super impressed, but I don't have time to full parse the thing right now. It is a bit surprising; if memory serves, one of the authors on this paper had a much better result in terms of neural network fact editing in the last year or two. This looks like a solid research idea, solid work, it didn't pan out, and to get it published they heavily overstated the conclusions (and then the university press release obviously bragged as much as it could).

Re:LLMs use a surprisingly simple mechanism to retrieve some... by ksenzee on Thursday March 28, 2024 @12:06PM (#39853318)
I think they're trying to say "equations in the form y = mx + b" without getting too technical.
Re:LLMs use a surprisingly simple mechanism to retrieve some... by 01HNNWZ0MV43FF on Thursday March 28, 2024 @11:52AM (#39853023)
Yeah I guess they mean one independent variable and one dependent variable

It rarely matters because if you had 2 dependent variables, you can just express that as 2 equations, so you might as well assume there's exactly 1 dependent and then only discuss the number of independent variables.

Re:LLMs use a surprisingly simple mechanism to retrieve some... by pb060 on Thursday March 28, 2024 @12:23PM (#39853665)
Aren’t functions and equations two different things?
Re:LLMs use a surprisingly simple mechanism to retrieve some... by olejorgenb on Thursday March 28, 2024 @12:02PM (#39853220)
I would think `x = 4` is considered an equation, yes?
Re:LLMs use a surprisingly simple mechanism to retrieve some... by pessimizer on Thursday March 28, 2024 @12:30PM (#39853815)
And linear at that: x = 0y + 4