News for hackers. Stuff that matters.
Re:Transformers as Support Vector Machines by noduerme on Sunday September 03, 2023 @04:44AM (#37368802)
Fuck, imagine how many doctoral theses I could've written every time I tweaked a few lines of code to try some abstract way of recombining outputs I didn't fully understand. I missed the boat. All this jargon is absolutely for show, though. Purely intended to create the impression that there's some kind of moat to the "discovery". There are much clearer ways to express "we fucked around with putting the outputs of this black box back into the inputs", but I guess that doesn't impress the rubes.
Re:Transformers as Support Vector Machines by ResearchCode on Sunday September 03, 2023 @05:24AM (#37368954)
You're not wrong. Applied ML articles are not worth reading.
Re:Transformers as Support Vector Machines by constantly on Sunday September 03, 2023 @10:26AM (#37370697)
I wouldn’t go this far, applied ML articles are my favorite articles. If you’re in the arena, it’s good to see things that other people have done from a practical perspective so you can ape it in your own work or not give it further consideration.
Re:Transformers as Support Vector Machines by kristopolous on Sunday September 03, 2023 @05:53AM (#37369086)
I really really wish this culture of expressing simple things in ornate ways would die. All it does is make knowledge less accessible
Re:Transformers as Support Vector Machines by tinco on Sunday September 03, 2023 @06:38AM (#37369272)
Of the excuse is true and the "ornate" language really is a dense representation of information then it should be fairly trivial to have an LLM agent unsummarize it.

There could be a webservice that offers a parallel track of layman's translations of any paper.

Re:Transformers as Support Vector Machines by heydenberk on Sunday September 03, 2023 @08:22AM (#37369824)
Sounds like you're referring to https://www.explainpaper.com/ :)
Re:Transformers as Support Vector Machines by noduerme on Sunday September 03, 2023 @08:30AM (#37369888)
An LLM won't tell you that the authors obfuscated it because they don't know what the fuck they did. You need a human for that.
Re:Transformers as Support Vector Machines by estebarb on Sunday September 03, 2023 @11:53AM (#37371385)
I haven't tried other models, but if you prompt a recent ChatGTP with "academic style" and ask it to "review and provide feedback" of a paragraph you wrote it will reword it using the most fancy, overselling words it can find. I liked to use it for improving grammar and style, but in later iterations ChatGTP started writing garbage...

I'm not sure if that is because training, feedback from users or an attempt to make usage is LLMs obvious to teachers.

Re:Transformers as Support Vector Machines by jimkoen on Sunday September 03, 2023 @05:58AM (#37369102)
People have to get their PhD's somehow... ;)
Re:Transformers as Support Vector Machines by noduerme on Sunday September 03, 2023 @08:23AM (#37369834)
I think this applies to everything right now. Papers like this are just ridiculous examples. In like, 6th grade I won second place at the LA county science fair for coding a simulation of a coyote's life in hypercard (with tons of graphs). Yay. Y'know what? That shit and those graphs would've been incomprehensible to the judges if it hadn't been written in plain language, in an attempt to make them understand what they were looking at. My entire career since has been an attempt to communicate and alleviate the pain points in communication between parties, by way of writing software that encapsulated their descriptions of what they needed. And likewise I never pretended to be smarter or know more than my clients did: Everything must be explained and comprehensible in normal people language. People need to know how shit works, especially if they're paying for it.

Or they should.

Or if they don't know and don't care, they're fucking negligent.

Especially if they say "wow that sounds smart, let's let these guys run our weapons program".

To your point, the reason this ornate language thrives and people get away with complacency about how their own systems work, boils down to a silent pact between managers and engineers to sweep everything under the rug out of laziness and ill-will. There's something blatantly mendacious and evil (in the banal way) about the agreement that managers approve black boxes which were approved by complex-sounding papers so that upper management can wash their hands of the results.

[edit] maybe I'm just bitter because I spent hours today pondering exactly how many engineers at Monsanto must have known about the dangers of the astroturf, and how many raised their hand, or hid behind a spreadsheet

https://frontofficesports.com/investigation-links-astroturf-...

Re:Transformers as Support Vector Machines by pedrosorio on Sunday September 03, 2023 @09:29AM (#37370272)
“Engineers who like to pretend to be mathematicians” - I heard once.
Re:Transformers as Support Vector Machines by kristopolous on Sunday September 03, 2023 @09:45AM (#37370387)
In math, this is mostly an English problem I think. Next time you find a Wikipedia math page to be an impenetrable wall of jargon, click the Wikipedia language tool and choose another language, any will do.

Then use Chrome's tool to machine translate the foreign language version back to English. I've found invariably this makes the article more coherent then the native English language Wikipedia math page.

It says something about the culture for sure.

Re:Transformers as Support Vector Machines by novalis78 on Sunday September 03, 2023 @10:41AM (#37370808)
This works for philosophers too. Try Schopenhauer (already easy to read in comparison) in English translation.
Re:Transformers as Support Vector Machines by thechao on Sunday September 03, 2023 @10:31AM (#37370735)
I know a lot of professional mathematicians who can't make their way through the Wikipedia articles of adjacent fields. The English entries appear to be written by the ghost of Bourbaki!
Re:Transformers as Support Vector Machines by consilient on Sunday September 03, 2023 @10:50AM (#37370882)
Some of them are: I've run into sections lifted straight of of Lang's Algebra several times.
Re:Transformers as Support Vector Machines by djmips on Monday September 04, 2023 @03:07AM (#37377914)
Since LLM are good at translating perhaps that's one of the reasons they seem to do a decent job of explaining things.
Re:Transformers as Support Vector Machines by CuriouslyC on Sunday September 03, 2023 @09:53AM (#37370444)
That's literally the entire field of philosophy after the ~18th century.
Re:Transformers as Support Vector Machines by FrustratedMonky on Sunday September 03, 2023 @11:12AM (#37371051)
Yep ~18 century Didn't Wittgenstein and/or Nietzsche say something similar. Words are in-adequate for communication, and all philosophy is playing with words.

But, Language is all we have to communicate, so guess we are stuck with it.

Re:Transformers as Support Vector Machines by TrackerFF on Sunday September 03, 2023 @10:45AM (#37370844)
"Not math-y enough"/ "Needs more math" is a very common feedback ML/AI researchers get when writing papers.

The other day I was watching a live-stream of a doctoral defense, as the thesis was quite relevant to my work.

So one of the committee members would really pick and criticize the math - ask questions like "You are supposed to be the bleeding edge on this topic, why was the math so simple? Did you research more rigorous theories to explain the math?" etc. (He was awarded the doctorate though)

So, I dunno, if that's how things are now - it makes sense to me that the authors go overboard with complicated notation, even if they could have written it much simpler. Probably makes the work seem more rigorous and legit.

Doesn't really take that much more time, and it covers your ass from "not rigorous enough" gotchas - though at the expense of readability.

Re:Transformers as Support Vector Machines by kristopolous on Sunday September 03, 2023 @11:00AM (#37370957)
Go read any article in the first 200 years or so of philtrans. There's lots of crucial science there written in a way that doesn't have the modern trappings of the form. It's good reading. Maybe some style perturbations borrowed from earlier eras would be good

https://www.biodiversitylibrary.org/bibliography/62536 menu on the right

Benjamin Franklin, Robert Boyle, Isaac Newton, Maxwell, Ohm and Volt - they're all there. If that style was good enough for them ...

Re:Transformers as Support Vector Machines by FrustratedMonky on Sunday September 03, 2023 @11:08AM (#37371021)
I wish also. When I was young and new, so much wasted time trying to parse the 'arcane' math that was really something simple bug dressed up as complicated to give it weight.
Re:Transformers as Support Vector Machines by mattew on Sunday September 03, 2023 @11:25AM (#37371137)
What I find entertaining/confounding is how difficult the abstracts to these new AI papers are to understand. It feels like academia is pushing this style, so it’s hard to blame the authors since they have to play the game.

For reference I have an undergrad degree in computer science, have been working professionally for 25 years, and am fairly data centric in my work.

I’m hoping when I run this through GPT4 to get an explanation for a mortal software developer something sensible comes out the other end.

Re:Transformers as Support Vector Machines by necroforest on Sunday September 03, 2023 @06:48AM (#37369325)
which jargon here is "just for show"?
Re:Transformers as Support Vector Machines by ftxbro on Sunday September 03, 2023 @09:30AM (#37370278)
> we show that over-parameterization catalyzes global convergence by ensuring the feasibility of the SVM problem and by guaranteeing a benign optimization landscape devoid of stationary points

does this mean 'an over-parameterized transformer problem is a convex svm problem'?

Re:Transformers as Support Vector Machines by dongecko on Sunday September 03, 2023 @10:10AM (#37370562)
I read it the same way as you did, or at least it's an approximation.

In general that's not really surprising. I remember discussions from some years ago about larger networks leading to smother loss surfaces.

Re:Transformers as Support Vector Machines by tensor on Sunday September 03, 2023 @10:59AM (#37370946)
The irony is that your "simplification" uses even more "jargon."

But yes, thats how I would read that, and I also see no issue at all with the language in the paper. These terms are used for precision, and have meaning to those in the field. Papers are written for other experts, not laymen.

Re:Transformers as Support Vector Machines by ftxbro on Sunday September 03, 2023 @11:09AM (#37371024)
OK but why they write "benign optimization landscape devoid of stationary points" instead of "convex" other than "just for show"? In my understanding it's not better for either audience experts or laymen. For experts it would be more clear to just say convex and they would know the implications, and if someone doesn't know what convex means they probably also aren't going to be on board with 'stationary points'. Also I'm not trying to pick on the authors I'm just trying to answer the question of which specific parts could be seen as 'just for show'.
Re:Transformers as Support Vector Machines by sdenton4 on Sunday September 03, 2023 @11:26AM (#37371141)
Because if you say convex, you better damned well have a proof of convexity...
Re:Transformers as Support Vector Machines by agent327 on Sunday September 03, 2023 @07:41AM (#37369589)
You think it's that easy, but, as frequently the case with transformers, I believe there's more here than meets the eye.
Re:Transformers as Support Vector Machines by gargablegar on Sunday September 03, 2023 @09:07AM (#37370105)
I studied using ML just over a decade ago. I actually compared MLPs to SVMs and had a similar thought to this. It does seem like there is a regression on understanding some of the fundamentals and older tools of the trade.

I guess everyone gets focused on the newer things.

Really does seem like people rediscovering older endpoints.

Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @10:25AM (#37370682)
There's been a huge flood of vanilla software engineers into ML, retconning it as "a subfield of computer science" (computability is a minor concern compared to the statistical underpinnings). They pretend to know the math because they can read the equations, then claim with utmost confidence that actually they're doing all the hard work in ML because they are experts in calling APIs and integrating into products, however useful or useless.
Re:Transformers as Support Vector Machines by jjtheblunt on Sunday September 03, 2023 @11:05AM (#37370999)
Retconning? What does that mean?
Re:Transformers as Support Vector Machines by chriswarbo on Sunday September 03, 2023 @11:33AM (#37371206)
It's a term from literature/fiction, where later installments try to explain something from earlier installments; usually in a way that's nothing like the original intent. Applying this to real history is more like "revisionism".
Re:Transformers as Support Vector Machines by ReactiveJelly on Sunday September 03, 2023 @02:24PM (#37372851)
"RETroactive CONtinuity", in addition to what the sibling comment said.

The Wikipedia top example is Sherlock Holmes dying in a fight with Moriarty and then coming back later when the author relented and decided to write more stories.

https://en.wikipedia.org/wiki/Retroactive_continuity

Re:Transformers as Support Vector Machines by jjtheblunt on Monday September 04, 2023 @12:16AM (#37377060)
(thanks to the sibling comments for explaining...)
Re:Transformers as Support Vector Machines by visarga on Sunday September 03, 2023 @11:35AM (#37371222)
There's a similar trend of accusing LLMs of not really understanding, being just pattern machines. Funny that a whole group of people get the same treatment.
Re:Transformers as Support Vector Machines by Q6T46nT668w6i3m on Sunday September 03, 2023 @11:49AM (#37371352)
You’re comparing a subjective claim to an objective claim. Let’s be clear, transformers do not “understand.”
Re:Transformers as Support Vector Machines by og_kalu on Sunday September 03, 2023 @02:41PM (#37373052)
They do in every practical, testable definition of the word.

"Transformers don't understand" is not an objective claim and in fact any attempt to objectify it leads to the opposite assertion.

Re:Transformers as Support Vector Machines by jprete on Sunday September 03, 2023 @12:58PM (#37371949)
I'm not sure what your point is. Many software engineers blatantly pattern-match and copy-paste code to try to get their own stuff to work without understanding what's really going on. This is a long-standing complaint in the industry.
Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @01:08PM (#37372032)
You will note the phrasing is tautological in nature and does not apply to all software engineers.
Re:Transformers as Support Vector Machines by consilient on Sunday September 03, 2023 @02:14PM (#37372740)
> "a subfield of computer science" (computability is a minor concern compared to the statistical underpinnings)

Computability theory is not all of computer science. It's just one subfield among many.

Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @06:28PM (#37375211)
I'm referring obliquely to a specific nitpick from select CS folks who argue that because the theoretical optimum is not computable in finite time/memory that the statistical basis for understanding ML is irrelevant.
Re:Transformers as Support Vector Machines by noduerme on Sunday September 03, 2023 @05:31PM (#37374819)
Really? Here I thought it was a flood of academics retconning neural net code into a "science" now that programmers had made it run in Python for them fast enough to be useful.
Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @06:25PM (#37375187)
As hacky as it ends up being in practice, there are some pretty solid theoretical fundamentals to the field of statistical learning.

The problem is the theory is constrained either to the micro-scale (individual layers/"simple" models, etc.) or to the supra-scale (optimization/learning theory, etc.).

Not much concrete can be said about the macro-scale (individual networks) in theoretical terms, only that empirically they seem tend toward the things the supra-scale theory says they should do.

The current controversy in the academia v engineers tussle is 1) what exactly do the empirical results imply and 2) how much does the theory really matter given the practical outcomes. The only thing the two sides broadly agree upon is that some amount of error will always exist because NNs can be broadly understood as lossy compression machines.

Re:Transformers as Support Vector Machines by Q6T46nT668w6i3m on Sunday September 03, 2023 @11:51AM (#37371360)
That’s science. You can’t expect everyone to know everything. It’s a preprint so this is the first opportunity to provide feedback.
Re:Transformers as Support Vector Machines by meltyness on Sunday September 03, 2023 @10:11AM (#37370569)
Or, optimistically, this is really how they think about these things, and you should simply be happy they're not trying to obfuscate their findings.
Re:Transformers as Support Vector Machines by thechao on Sunday September 03, 2023 @10:17AM (#37370625)
Watching the AI community rediscover automatic differentiation 20+ years after the field was considered "mature" was equal parts frustrating & fascinating. The frustration was them rewriting the history of discovery, but without any sort of sense or rigor ... and it was also the most fascinating!
Re:Transformers as Support Vector Machines by AndrewKemendo on Sunday September 03, 2023 @10:28AM (#37370709)
This is indeed the frustration

I'm waiting for some fresh group of grad students to make a breakthrough using a reinvented version of Pearls "Do" calculus or maybe they make some narrow breakthrough using BayesNets and everyone geeks out on those for a while

*I do think transformers (much like ff networks + backprop from 2012-2018) are probably a lasting software architecture for inference applications until we come up with new hardware, and move beyond GPU focused computing

It's exciting to see it all working, but disheartening how a-historical this last few years has been in AI - with the exception of Brooks, Sutton and a few other greybeards in the field who say similarly

Re:Transformers as Support Vector Machines by barelyauser on Monday September 04, 2023 @07:43AM (#37379548)
Scarcity is not a myth.
Re:Transformers as Support Vector Machines by AndrewKemendo on Thursday September 07, 2023 @04:28PM (#37425095)
There is no scarcity of fundamental human needs of water, food, shelter and love.

The only reason someone lacks them is because someone else is hoarding them.

This is well established in global trade metrics.

Re:Transformers as Support Vector Machines by capableweb on Sunday September 03, 2023 @11:26AM (#37371147)
The funny thing is that this constantly happens in every field ever, humans truly excel at repeating history without learning from the past.

Another example:

- HTML served by static file servers

- HTML generated by backend

- HTML enhanced with small JS snippets

- HTML generated by frontend, but served by backend

- Go to step one, not learning why anyone moved on from the previous method

Re:Transformers as Support Vector Machines by joaogui1 on Sunday September 03, 2023 @01:27PM (#37372200)
Programmers excel at it, you don't see that happening much with Physics, Chemistry, Math, Material Sciences etc Funnily enough it has also been observed before, by Alan Kay comparing programming to a pop culture "In the last 25 years or so, we actually got something like a pop culture" https://queue.acm.org/detail.cfm?id=1039523#:~:text=In%20the...
Re:Transformers as Support Vector Machines by mr_toad on Sunday September 03, 2023 @09:17PM (#37376222)
Poor training, poor communication, & knowledge not being curated.

When then best method of getting advice on the internet is to post the wrong answer you know the system is broken.

Re:Transformers as Support Vector Machines by Q6T46nT668w6i3m on Sunday September 03, 2023 @11:47AM (#37371327)
It’s a preprint on arXiv, not the Magna Carta.
Re:Transformers as Support Vector Machines by maf12 on Sunday September 03, 2023 @02:25PM (#37372864)
I think the main motivation in ml theory that touches current SOTA is not "expressing simple ideas with a jargon for show". Jargon is necessary, as much as some (mostly very practical) engineers or software people cannot see it due to how unnecessary it seems to them (as they are used to practically and quickly express themselves). It's a jargon for the mathematics of machine learning, which is pretty unstandardized so to speak. So you need to define yourself. And without a jargon and clear proofs, what you do is just brainstorming at most. The value of such work is that their statement is pretty clear, proved and contain hypotheses which can be tested by the future papers.

Here is an example: to explain the existence of adversarial example, there are 2 suggestions without a jargon: 1) that the decision boundary is too nonlinear, 2) that the decision boundary is too linear. Both of these explanations contradict and stated without any real proof and unfortunately can be widely heard in most of the adversarial example papers. If we were to have clear formulations of these two statements, we could have tested both of these claims but unfortunately the papers that suggested these theories didn't put effort for defining a jargon and putting their suggestion as a clear-formal statement.

Re:Transformers as Support Vector Machines by r-zip on Monday September 04, 2023 @10:14AM (#37380672)
What are you talking about? Did we read the same abstract?