News for hackers. Stuff that matters.
Re:Transformers as Support Vector Machines by gargablegar on Sunday September 03, 2023 @09:07AM (#37370105)
I studied using ML just over a decade ago. I actually compared MLPs to SVMs and had a similar thought to this. It does seem like there is a regression on understanding some of the fundamentals and older tools of the trade.

I guess everyone gets focused on the newer things.

Really does seem like people rediscovering older endpoints.

Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @10:25AM (#37370682)
There's been a huge flood of vanilla software engineers into ML, retconning it as "a subfield of computer science" (computability is a minor concern compared to the statistical underpinnings). They pretend to know the math because they can read the equations, then claim with utmost confidence that actually they're doing all the hard work in ML because they are experts in calling APIs and integrating into products, however useful or useless.
Re:Transformers as Support Vector Machines by jjtheblunt on Sunday September 03, 2023 @11:05AM (#37370999)
Retconning? What does that mean?
Re:Transformers as Support Vector Machines by chriswarbo on Sunday September 03, 2023 @11:33AM (#37371206)
It's a term from literature/fiction, where later installments try to explain something from earlier installments; usually in a way that's nothing like the original intent. Applying this to real history is more like "revisionism".
Re:Transformers as Support Vector Machines by ReactiveJelly on Sunday September 03, 2023 @02:24PM (#37372851)
"RETroactive CONtinuity", in addition to what the sibling comment said.

The Wikipedia top example is Sherlock Holmes dying in a fight with Moriarty and then coming back later when the author relented and decided to write more stories.

https://en.wikipedia.org/wiki/Retroactive_continuity

Re:Transformers as Support Vector Machines by jjtheblunt on Monday September 04, 2023 @12:16AM (#37377060)
(thanks to the sibling comments for explaining...)
Re:Transformers as Support Vector Machines by visarga on Sunday September 03, 2023 @11:35AM (#37371222)
There's a similar trend of accusing LLMs of not really understanding, being just pattern machines. Funny that a whole group of people get the same treatment.
Re:Transformers as Support Vector Machines by Q6T46nT668w6i3m on Sunday September 03, 2023 @11:49AM (#37371352)
You’re comparing a subjective claim to an objective claim. Let’s be clear, transformers do not “understand.”
Re:Transformers as Support Vector Machines by og_kalu on Sunday September 03, 2023 @02:41PM (#37373052)
They do in every practical, testable definition of the word.

"Transformers don't understand" is not an objective claim and in fact any attempt to objectify it leads to the opposite assertion.

Re:Transformers as Support Vector Machines by jprete on Sunday September 03, 2023 @12:58PM (#37371949)
I'm not sure what your point is. Many software engineers blatantly pattern-match and copy-paste code to try to get their own stuff to work without understanding what's really going on. This is a long-standing complaint in the industry.
Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @01:08PM (#37372032)
You will note the phrasing is tautological in nature and does not apply to all software engineers.
Re:Transformers as Support Vector Machines by consilient on Sunday September 03, 2023 @02:14PM (#37372740)
> "a subfield of computer science" (computability is a minor concern compared to the statistical underpinnings)

Computability theory is not all of computer science. It's just one subfield among many.

Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @06:28PM (#37375211)
I'm referring obliquely to a specific nitpick from select CS folks who argue that because the theoretical optimum is not computable in finite time/memory that the statistical basis for understanding ML is irrelevant.
Re:Transformers as Support Vector Machines by noduerme on Sunday September 03, 2023 @05:31PM (#37374819)
Really? Here I thought it was a flood of academics retconning neural net code into a "science" now that programmers had made it run in Python for them fast enough to be useful.
Re:Transformers as Support Vector Machines by uoaei on Sunday September 03, 2023 @06:25PM (#37375187)
As hacky as it ends up being in practice, there are some pretty solid theoretical fundamentals to the field of statistical learning.

The problem is the theory is constrained either to the micro-scale (individual layers/"simple" models, etc.) or to the supra-scale (optimization/learning theory, etc.).

Not much concrete can be said about the macro-scale (individual networks) in theoretical terms, only that empirically they seem tend toward the things the supra-scale theory says they should do.

The current controversy in the academia v engineers tussle is 1) what exactly do the empirical results imply and 2) how much does the theory really matter given the practical outcomes. The only thing the two sides broadly agree upon is that some amount of error will always exist because NNs can be broadly understood as lossy compression machines.

Re:Transformers as Support Vector Machines by Q6T46nT668w6i3m on Sunday September 03, 2023 @11:51AM (#37371360)
That’s science. You can’t expect everyone to know everything. It’s a preprint so this is the first opportunity to provide feedback.