News for hackers. Stuff that matters.
Re:The AI Crowd Is Mad by rco8786 on Wednesday February 01, 2023 @08:47AM (#34610306)
Is the author conflating the cost to train an AI model with the cost to use the resulting models, here?

I can certainly believe that model training cost is exponential as the # of parameters goes exponential. But that is a one time(-ish) cost relative to actually using those models.

Or am I completely off base here?

Re:The AI Crowd Is Mad by ahoho on Wednesday February 01, 2023 @09:12AM (#34610644)
Inference costs are non-trivial, and I wouldn’t be surprised if the cost of running ChatGPT (given the 3M/day figure) has surpassed that of training it. Without optimizations, training only uses ~3 times the memory as inference, so exponential parameter/cost scaling still affects both.

There’s ongoing research to reduce the computational costs of inference, but to my knowledge they only offer linear improvements (although I wouldn’t bet against more substantial reductions in the near future, particularly as these techniques are compounded).

Re:The AI Crowd Is Mad by gizmo on Wednesday February 01, 2023 @09:54AM (#34611259)
OpenAI has stated that their costs per query are a few cents. Not nothing, but nothing outrageous either. The cost will go down to < 1c within a couple of years I'm sure. The cost may be prohibitive to use LLMs for absolutely everything, but the universe of applications where companies can afford to spend 1c on an LLM answer is very large.
Re:The AI Crowd Is Mad by smoldesu on Wednesday February 01, 2023 @10:05AM (#34611439)
> the universe of applications where companies can afford to spend 1c on an LLM answer is very large.

I hear this a lot, but don't see it very often in practice. GPT and it's ilk are coming up on 3 years old now, and our most novel application thus far is the same textbox + response that Talk to Transformer had. That's sad.

I've heard the sell before ("imagine AI spreadsheets!") but I don't think people are willing to pay for answers that are regularly wrong in the long run. If your bridge only works half the time, people probably won't be inclined to pay your toll anymore.

Re:The AI Crowd Is Mad by gizmo on Wednesday February 01, 2023 @10:37AM (#34611965)
Today, the typical customer support experience involves waiting for hours/days to get a response that is useless and/or wrong. LLM powered customer support software can be transformational, and this is just one example.
Re:The AI Crowd Is Mad by smoldesu on Wednesday February 01, 2023 @11:15AM (#34612641)
LLM-powered customer support can also be useless and wrong. Even in an ideal case, you're still retaining some customer support employees and only displacing the pitifully-paid call-center workers that staff the current Chat-As-A-Service offerings.

Until I see something like this rolled out with widespread success, I'm gonna doubt it. The second someone puts an AI agent on their website, it's a race to get the brand to endorse the most abhorrent thing possible. Then what?

Re:The AI Crowd Is Mad by gizmo on Wednesday February 01, 2023 @12:20PM (#34613909)
You don't want the LLM to be completely unsupervised. The goal is assistance, not the equivalent of full self-driving. It takes a few seconds for a support tech to approve a response that would take 15 minutes to compose by a human. That's where the value is. You automate the rote repetitive work so the interesting intelligent work remains.
Re:The AI Crowd Is Mad by smoldesu on Wednesday February 01, 2023 @01:48PM (#34615512)
"The value" is in the self-driving part though. The support tech wants the good answer, not the part where they refresh ChatGPT 3 times because the output is unintelligible. If you're going to pay a human to do the job either way, I'd bet that a skilled technician out-performs an AI model operated by a classifier employee.
Re:The AI Crowd Is Mad by layer8 on Wednesday February 01, 2023 @09:12AM (#34610643)
In the long run, a model that isn’t regularly updated with new data won’t be very viable for many purposes. The (re)training cost therefore will have some relevance.
Re:The AI Crowd Is Mad by fock on Wednesday February 01, 2023 @09:08AM (#34610569)
Well, the number of parameters is directly related to the size of the machine to keep the model around. And so while you can host GPT-2 on a single consumer GPU, for ChatGPT you allegedly need 4-8-10k-a-piece GPUs. Which also consume 8-16 the power of your consumer GPU. So if you continue that, the cost to society will be too high to bear for this kind of games. (and in my eyes already is, as the outputs of large generative models are 100% used to generate more stuff to consume for the same amount of already information-overloaded people)
Re:The AI Crowd Is Mad by ryan93 on Wednesday February 01, 2023 @09:25AM (#34610851)
A single international flight literally consumes a swimming pools worth of fuel. a few GPUs is nothing especially as we transition to wind and solar.
Re:The AI Crowd Is Mad by fock on Wednesday February 01, 2023 @09:37AM (#34611019)
well, then stop flying and the useless content-creation and let's get back to our "few GPUs" for consumption, when living, food and flying have transitioned to wind and solar. Then people can start setting up their text-based Holodeck-precursor consuming 1kW during usage.

Just be clear: the alternative to letting a future GPT-derivative write your novel is to go to the library and pick a book. That works now and has negligible emissions. A future GPT (currently you won't get a consistent long form text) might do the same - optimistically it will do it on only 2-3kWh of energy (which amounts to the cost of creating a physical book). Granted, you only can sustain a human on that for a day, but then the humans don't go away because of AI.

Conclusion to me: the things are fascinating and so are H-bombs...

Question: which kind of time do you expect people to spend with these models? Which human needs will they satisfy? How will their energy consumption then still be negligible?

Re:The AI Crowd Is Mad by ryan93 on Wednesday February 01, 2023 @01:10PM (#34614819)
This reads like you ran it through a translator.
Re:The AI Crowd Is Mad by fock on Wednesday February 01, 2023 @04:16PM (#34617892)
so? translation is not the point of this models anymore.
Re:The AI Crowd Is Mad by tomrod on Wednesday February 01, 2023 @10:09AM (#34611504)
> Well, the number of parameters is directly related to the size of the machine to keep the model around.

I bet this has some potential.

Re:The AI Crowd Is Mad by habibur on Wednesday February 01, 2023 @08:52AM (#34610377)
I pretty much always wanted to know the cost of running those models, not training those.

That should be order of magnitude less.

Re:The AI Crowd Is Mad by jackmott42 on Wednesday February 01, 2023 @09:01AM (#34610487)
stable diffusion will generate an image in ~30 seconds on an NVidia 3060, so the cost is on the order of cents per image, power use on the order of 170 watts during the generation, or 0.0014166666667 kwh. @ around 5 cents per kwh
Re:The AI Crowd Is Mad by rileymat2 on Wednesday February 01, 2023 @09:04AM (#34610523)
You also have to calculate the life expectancy of the card against 30 seconds of its life as an ongoing cost.
Re:The AI Crowd Is Mad by MagnumOpus on Wednesday February 01, 2023 @09:31AM (#34610949)
Which is also on the order of 0.01c per query (answering 5 million queries in the lifetime of a $500 card).
Re:The AI Crowd Is Mad by smulc on Wednesday February 01, 2023 @09:10AM (#34610601)
Just ask ChatGPT what the cost is? ;)
Re:The AI Crowd Is Mad by npongratz on Wednesday February 01, 2023 @09:19AM (#34610758)
Sure, but just remember that it can't do math, so any mathy answer it gives is just bullshit.