State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months, huge amounts of electrical power, and expensive-to-create proprietary datasets. Not to mention large team of highly-in-demand experts with very expensive salaries.

Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.

(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)

And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.

> State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months

This is changing very rapidly. You don’t need that anymore

https://twitter.com/karpathy/status/1661417003951718430?s=46

There’s an inverse Moore’s law going on with compute power requirements for AI models

The required compute power is decreasing exponentially

Soon (months, maybe a year), people will be training models on their gamer-level GPUs at home, maybe even on their computer CPUs

Plus all the open and publicly available models both on HuggingFace and on GitHub

Roll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that.

For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc.

Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.

It doesn’t matter

The point is that this is unstoppable

They aren’t saying that AI needs to stop existing, just that more effort needs to be put towards regulating it, monitoring it, aligning it, etc.

And what I’m saying is that it doesn’t matter and it is unstoppable

"nico" has spoken

You can't build gpt-3 or gpt-4 in a basement, and won't be able to without several landmark advancements in AI or hardware architectures. The list of facilities able to train a GPT-4 in <5 years can fit on postcard. The list of facilities producing GPUs and AI hardware is even shorter. When you have bottlenecks you can put up security checkpoints.

I'm very confused that this point is being ignored so heavily on HN of all places. If tomorrow ASML and TSMC are struck by a meteor, or indeed controlled/sanctioned, it would take either the US or China spending trillions and cost many years to rebuild this. It's not something that can be done in secret either.