News for hackers. Stuff that matters.
Re:ChatGPT Now Losing Users by damnesian on Thursday July 13, 2023 @09:46AM (#36709040)
>Have we hit peak AI?

AI is at the same point cinema was at when pioneers of the genre showed short films in "salons" designed to frighten people with simple tricks.

This is hardly the peak. We're just at the beginning.

Re:ChatGPT Now Losing Users by orwin on Thursday July 13, 2023 @10:50AM (#36710093)
AI development is different than most CS fields, as you need a solid mathematical base to build a "new discovery", and a lot of computation to use it efficiently.

I'm 100% certain it's not the AI peak, but i'm also confident that we are close to the local maxima, at least for transformer architecture. Maybe the following improvement will be in chaining architecture, or a brand new technique, or maybe progress will be made in parallel fields (even smaller processors!).

Charitably, i'll decide that the question meant "Have we hit peak GPT" and in this case, this is a legitimate question.

Re:ChatGPT Now Losing Users by unshavedyak on Thursday July 13, 2023 @10:58AM (#36710228)
I'm most interested in refined training sets, refined compression, etc. Where do you suspect the cap is there?

I say that because i find GPT4 extremely useful. Even if GPT4 (in it's hypothetical un-nerfed state some would argue)* is the peak LLM i think we have a ton of room to grow even within that peak.

Locally running larger context windows, multiple domain experts in parallel and hooked up to a local repository of available IO actions would give a very significant "advancement" within the same conceptual "peak LLM". Now will we be able to get there on consumer hardware? Not idea. However i don't think i need more than GPT4 to find the whole technology extremely useful.

Moderately cheap to run large context windows[1] entirely locally is the "next big thing" for me. I get a fair bit of value from GPT4 and Phind. The technology has sold me already.

[1]: I should amend, i'm speaking loosely about context windows. What i mean is the ability to feed it refined datasets locally to have it's "local knowledge" change, which may be less about context windows and more about refined training. I'll leave that to the experts to debate, i am assuredly nothing close.

Re:ChatGPT Now Losing Users by orwin on Thursday July 13, 2023 @11:35AM (#36710880)
> I say that because i find GPT4 extremely useful. Even if GPT4 (in it's hypothetical un-nerfed state some would argue)* is the peak LLM i think we have a ton of room to grow even within that peak.

Yes. sorry my point wasn't well formulated (i'm not an expert either: i've trained LLMs on receipts in 2016, and quickly decided i hated that). I'll try doing it better by dividing the two idea i put in that post:

- i think that GPT4 is probably close the peak generalist GPT, meaning that maybe Bard can surpass it (or a chatGPT4.5 or something) but the improvement won't be stellar.

- If future improvement won't come from the transformer architecture itself (i absolutely can be wrong on that), it'll come from what we interface with it.

Absolutely context windows will probably a great way to have "specialists" transformers, but alos linking it with different "kind" of AI might push it too (i've read a post from Wolfram a while ago that sold me on interlinking different AI models/languages/state machine). I'm sure specialist or even large users haves tons of ideas.

I got some value from GPT3.5, enough for me to buy access to GPT4. with a caveat: i'm pretty sure that for work-related questions i lost a bit more time than i "gained" overall. because my mastery of the technologies we use and very specific needs that GPT cannot know. But in the last three weeks or so, i'm pretty sure it was overall a net benefit at work, as i know better when i should use it.

Re:ChatGPT Now Losing Users by CuriouslyC on Thursday July 13, 2023 @11:01AM (#36710292)
I don't think we'll need to completely replace the transformer, we just need better, more annotated data sets and a really good solution to large context windows. Data set annotation has a huge impact on model output, you can see this for yourself by comparing SDXL to Stable Diffusion 1.5, the model is larger but the real juice comes from using aesthetics scores for images during training to bias output towards things that look good - compare it to other large models like DALL-E or Imagen and you'll see parameter size is not the most important thing.

With larger context windows, we can use traditional search algorithms to cherry pick an educational context for the LLM for few shot learning. If that's paired with more, better and annotated data, that will take us quite far.

Re:ChatGPT Now Losing Users by YeGoblynQueenne on Thursday July 13, 2023 @10:54AM (#36710152)
We're not at the beginning. AI research started in 1956 with the Dartmouth workshop [1]. We've already had several cycles of boom and bust and at least two AI winters already [2].

Btw, there was a sci-fi novel where the plot took place on a planet where winter came in periods too long for the natives of the planet to remember (they had short lifespans, or they had limited memories, I er, why, I can't remember!). I don't remember the title of the novel or the author and I've been trying to find it. I thought it was something by Ursula K. Le Guin, but it doesn't sound like it.

And no, it's not Game of Thrones. I think GoT took the idea of long-coming winters from the earlier novel, or it's just a coincidence (or cryptomnesia).

Anyway I'd appreciate if someone knows what I'm talking about and lets me know.

___________

[1] https://en.wikipedia.org/wiki/Dartmouth_workshop

[2] https://en.wikipedia.org/wiki/AI_winter

Re:ChatGPT Now Losing Users by grumbel on Thursday July 13, 2023 @11:28AM (#36710744)
BingChat says it's Helliconia[1] by Brian W. Aldiss.

Nightfall and the movie Pitch Black also go in that direction.

[1] https://en.wikipedia.org/wiki/Helliconia

Re:ChatGPT Now Losing Users by YeGoblynQueenne on Thursday July 13, 2023 @03:29PM (#36714764)
Thanks!

Unforutnately Helliconia is not the one. I've read quite a bit of Brian Aldiss, but not that one. I think I would have remembered reading the Winter book in particular, judging from the discussion of the "Wheel of Kharnabar" in Wikipedia:

>> The Wheel is an extraordinary revolving monastery/prison built into a ring-shaped tunnel with a single entrance and exit, powered entirely by the efforts of the prisoners pulling it along by means of chains set into the outer wall. Once a prisoner enters a cell of the Wheel, it is impossible for him to leave until its full ten-year rotation has passed.

That's the kind of thing that sticks in my head. Anyway I remember bits of the plot and there wasn't so much intrigue and armies and whatnot. The native society was a primitive society that lived in yurts or tepees and didn't have much technology. I think they were supposed to be less intelligent than humans.

I don't think I'll ever forget Nightfall! :)

Re:ChatGPT Now Losing Users by dri_ft on Thursday July 13, 2023 @11:08AM (#36710402)
>a planet where winter came in periods too long for the natives of the planet to remember

Not Asimov's Nightfall?

Re:ChatGPT Now Losing Users by YeGoblynQueenne on Thursday July 13, 2023 @03:29PM (#36714769)
Nope. Winter, not night-time :)
Re:ChatGPT Now Losing Users by moffkalast on Thursday July 13, 2023 @11:02AM (#36710297)
We're not at the beginning, but we're not close to the end either.

The fact that organic brains exist proves the baseline of what's fundamentally possible and at very low power levels too if optimized well. We haven't reached that yet, and it's impossible to say what the upper limit is.

LLMs might not get there as an end-to-end solution but they'll will certainly be a significant part of one.

Re:ChatGPT Now Losing Users by stubybubs on Thursday July 13, 2023 @04:39PM (#36715664)
> I thought it was something by Ursula K. Le Guin, but it doesn't sound like it.

The Left Hand of Darkness is set on a planet with perpetual winter, though it somewhat warms up at times. Maybe that's what you're thinking of?

Re:ChatGPT Now Losing Users by rootusrootus on Thursday July 13, 2023 @11:04AM (#36710326)
I don't know, maybe this is the peak of this kind of AI. It seems like a lot of what drives this boom in LLMs isn't fundamental improvements in the technology, but rather the easy availability of enough computing power to do them. It's plausible that we're going to hit a wall abruptly and that the next step towards true AI will be down a different path.