A100s aren't magical pixie dust. China aren't developing their own silicon[1] for funsies.

This is all mathematics. Ignoring the fact that the entirety of what is required to build GPT-4 is easily remembered by a single human being (i.e. adversary), it could also be rediscovered. America doesn't have some secret sauce that nobody else has. We have a time lead, and that is it.

Keeping the details secret and under control is like trying to keep anyone from discovering that the sky is blue.

A 6 months freeze on AI research in America? China would relish it. Hell, who's to say this isn't their idea?

[1]: https://www.tomshardware.com/news/moore-threads-unveils-chun...

It reminds me of the crypto restrictions in the 90s. That just led to a few people outside the US becoming wealthy by selling tech unfettered by US crypto restrictions, like Mark Shuttleworth, now of Canonical/Ubuntu fame.

You can buy an A100 on eBay right now. If I had a few hundred grand I could have an A100 x8 box delivered to my house tomorrow.

You could probably run a model but not train a new one, especially one bigger than GPT-4. You'd need ten of thousand of A100 as a rough estimate.

I suspect some of us are going to be really disappointed when the actual parameter count for GPT-4 is discovered.

Has anyone considered the opposite possibility around the closed nature of OpenAI's work? I.e. Instead of worrying that someone will reproduce their 10k+ GPU GPT-4 training super farm, maybe the opposite happened - They discovered a way to increase performance while simultaneously reducing the required parameter counts and/or hardware.

Would that realization be more or less dangerous with this A100 constraint in mind?

> They discovered a way to increase performance while simultaneously reducing the required parameter counts and/or hardware.

In that case, why would they be so severely limiting access? If GPT4 was cheaper to run and better, surely they would have immediately transitioned away from 3.5?

(1) Some people speculate that GPT-4 is actually an ensemble of models selected by a supervisory model based on response timings.

(2) GPT4All is using improved training data with LoRA and GPT-J to build something that competes with GPT-4 at a fraction of the size and parameter count.

(3) The emergence of new capabilities strongly corresponds to parameter count. Transformers with 6.7B parameters are apparently where the first such emergence occurs associated with a global structural change in the hidden layers (the so-called “phase change”). There will be a strong incentive to train larger models to identify additional capabilities.

The first and second observations imply that federal regulation of A100-class GPUs would be ineffective. However, the third observation suggests that it would be highly effective at preventing untrusted parties from discovering novel and potentially more disruptive capabilities.