News for hackers. Stuff that matters.
Re:The bigger-is-better approach to AI is running out of road by TeMPOraL on Saturday June 24, 2023 @06:03PM (#36463065)
How much general "thinking"[0] would you want those "tight little specialist models" to retain? I think that cramming "all of the web" is actually crucial for this capability[1], so at least with LLM-style models, you likely can't avoid it. The text in the training data set doesn't encode just the object-level knowledge, but indirectly also higher-level, cross-domain and general concepts; cutting down on the size and breadth of the training data may cause the network to lose the ability to "understand"[0].

--

[0] - Or "something very convincingly pretending to think by parroting stuff back", if you're closer to the "stochastic parrot" view.

[1] - Per my hand-wavy hypothesis that the bulk of what we call thinking boils down to proximity search in extremely high-dimensional space.

Re:The bigger-is-better approach to AI is running out of road by nathan_compton on Saturday June 24, 2023 @07:15PM (#36463563)
This is basically my view as an extensive gpt4 user. It doesn't think/is very stupid, but has such fantastic recall for common patterns that its quite useful. Absolutely none of the smaller models even come close, including GPT3.5.
Re:The bigger-is-better approach to AI is running out of road by jiggawatts on Saturday June 24, 2023 @07:41PM (#36463759)
Even just for spell checking, having a ton of general knowledge helps. Knowing the correct spelling of product names, British vs American English, context-specific alternate spellings, etc…
Re:The bigger-is-better approach to AI is running out of road by thfuran on Sunday June 25, 2023 @09:08AM (#36467836)
Can you not just tell it not to muck with proper nouns?
Re:The bigger-is-better approach to AI is running out of road by jiggawatts on Sunday June 25, 2023 @07:15PM (#36473645)
You can, but then it won't correct misspellings of proper nouns!

The idea is for it to automatically detect the "language" of each word based on the context and its own understanding of the world.

E.g., the following sentence:

"We deployed windows server data centre 2022 into our data center, which has no windows for physical security."

Will be corrected by GPT-4 to the following:

"We deployed Windows Server Datacenter 2022 into our data centre, which has no windows for physical security."

Notice that it combined "data" and "centre" into "Datacenter" and it corrected the second "center" into "centre", which is the British/Australian spelling of the word. It also correctly capitalised only the first use of the word "windows", etc...

That requires a level of understanding that GPT 3.5 just barely has, and no ordinary grammar checker tool has.

Re:The bigger-is-better approach to AI is running out of road by jstarfish on Saturday June 24, 2023 @07:44PM (#36463772)
There are boundaries that can be defined, but I'm unclear on what those should be. I'm thinking breaking it down by semantic domains, but not at the exclusion of others altogether-- just heavily weigh certain domains in favor of others. Kinda like how StackOverflow is broken down by domain-- each subdomain could be a LORA or something.

General purpose models containing significant overlap between Project Gutenberg and Github are unnecessary and don't scale. Moby Dick has little to do with C++ unless you're creating art for novelty's sake. This is entirely speculative, but I'm convinced ChatGPT is faking the appearance of a single oracle while delegating requests to specialized models under the hood. It scales better and makes sense than trying to serve a 1T model to address everybody's banal questions.

Like, at its core, for people who only want to write literature, give them a model with underweighed programming-related corpora. Writers don't need it, will never use it, and that space could be filled with training content relevant to literature. Anything else results in expensive, unscalable solutions or jack-of-all-trades, master-of-none outcomes.

In recent usage, GPT3.5 helped me hack my way through writing Pester tests for Powershell scripts for the first time, and I mean hack-- there were a lot of assumptions it made and things it got wrong. GPT4 did a much better job, but I couldn't help but think 3.5 probably has a ton of other training data in it that detracts from the specialization I needed from it in that context. For coding help, you don't want to ask some random librarian who occasionally recommends resources that don't exist; you ask someone who specializes in coding and trust they have familiarity with that domain.

Re:The bigger-is-better approach to AI is running out of road by seydor on Sunday June 25, 2023 @12:45AM (#36465203)
They can't reason yet, but they can extrapolate, but it cant check if its extrapolation is reasonable. Reasoning is not baked into the architecture of a GPT model. It seems it would be an entirely different type of model
Re:The bigger-is-better approach to AI is running out of road by imtringued on Sunday June 25, 2023 @07:27AM (#36467090)
And that is exactly why the current generation of AI is the same as all the other generations. It is a bruteforce attempt to solve something that shouldn't need brute force.
Re:The bigger-is-better approach to AI is running out of road by TeMPOraL on Sunday June 25, 2023 @07:51AM (#36467245)
I used to believe that, but I'm no longer convinced. At least when you're trying to approximate the way humans think, and especially human language, there may not be a simpler way - our brains are themselves a product of randomness, and are unlikely to factor nicely into theoretically clean components. It might be computationally cheaper to repeat the process than trying to figure it out and encode in analytical form.
Re:The bigger-is-better approach to AI is running out of road by kaliqt on Monday June 26, 2023 @12:47PM (#36482241)
I think you two are both right in the most obvious of ways... it needs to trained on a ton of data so it can comprehend better, and when it needs to fact-check or search for anything concrete, it should hit up domain experts and databases.