All of us together do.

I saw the nerfing of GPT in real time: one day it was giving me great book summaries, the next one it said that it couldn't do it due to copyright.

I actually called it in a comment several months ago: copyright and other forms of control would make GPT dumb in the long run. We need an open source frontier less version.

There is a lot of randomness involved, are you sure it wasn’t just chance? If you try again it might work

So I thought I was getting great book summaries (from GPT 3.5, I guess) for various business books I had seen recommended, but then out of curiosity one day I asked it questions about a fiction book that I've re-read multiple times (Daemon by Daniel Suarez)... and well now I can say that I've seen AI Hallucinations firsthand:

https://chat.openai.com/share/d1cdd811-edc9-4d55-9cc1-a79215...

Not a very scientific or conclusive test to be sure, but I think I'll stick with using ChatGPT as my programming-rubber-ducky-on-steroids for now :)

Can't post this link enough: https://www.openpetition.eu/petition/online/securing-our-dig...

For now there is no other way to train models than the huge infrastructure. CERN have a tendency to provide results for the money spend and they have experience in building the infrastructures for sure.

I think a lot of people are unaware that these models have an enormous human training component performed through companies such as Amazon Mechanical Truk and dataannotation.tech. Called Human Intelligence Tasks, a large number of people have been working in this area for close to a decade. Dataannotation Tech claims to have over 100k workers. From Cloud Research,

"How Many Amazon Mechanical Turk Workers Are There in 2019? In a recent research article, we reported that there are 250,810 MTurk workers worldwide who have completed at least one Human Intelligence Task (HIT) posted through the TurkPrime platform. More than 226,500 of these workers are based in the US."

Here's an account of a person in Africa that helped train (wading thru gnarly explicit content in the process): https://www.bigtechnology.com/p/he-helped-train-chatgpt-it-t...

This reads like lawsuit bait.

Another thing that people don't know is that a lot of the safe-ified output is hand crafted. Part of "safety" is that a human has to identify the offensive content, decide what's offensive about it, and write a response blurb to educate the user and direct them to safety.

They don't want to know how the sausage is made.

How feasible would it be out crowdsource the training? I.e. thousands of individual macbooks training a small part of the model and contributing to the collective goal

Currently, not at all. You need low latency, high bandwidth links between the GPUs to be able to shard the model usefully. There is no way you can fit an 1T (or whatever) parameter model on a MacBook, or any current device, so sharding is a requirement.

Even if it that problem disappeared, propagating the model weight updates between training steps poses an issue in itself. It's a lot of data, at this size.

What if you split up the training down to the literal vector math, and treated every macbook like a thread in a gpu, with just one big computer acting as the orchestrator?

You would need each MacBook to have an internet connection capable of multiple terabytes per second, with sub millisecond latency to every other MacBook.

You could easily fit a 1T parameter model on a MacBook if you radically altered the architecture of the AI system.

Consider something like a spiking neural network with weights & state stored on an SSD using lazy-evaluation as action potentials propagate. 4TB SSD = ~1 trillion 32-bit FP weights and potentials. There are MacBook options that support up to 8TB. The other advantage with SNN - Training & using are basically the same thing. You don't have to move any bytes around. They just get mutated in place over time.

The trick is to reorganize this damn thing so you don't have to access all of the parameters at the same time... You may also find the GPU becomes a problem in an approach that uses a latency-sensitive time domain and/or event-based execution. It gets to be pretty difficult to process hundreds of millions of serialized action potentials per second when your hot loop has to go outside of L1 and screw with GPU memory. GPU isn't that far away, but ~2 nanoseconds is a hell of a lot closer than 30-100+ nanoseconds.

Edit: fixed my crappy math above.

That's been done already. See DeepSpeed ZeRO NVMe offload:

https://arxiv.org/abs/2101.06840

FWIW there are current devices that could fit a model of that size. We had servers that support TBs of RAM a decade ago (and today they're pretty cheap, although that much RAM is still a significant expense).

I have an even more of a stretch question.

What pieces of tech would need to be invented to make it possible to carry a 1T model around in a device the size of an iPhone?

Yes, someone revive Xgrid!

Whoa didn't know about this, cool

I once used a crowdsourcing system called CrowdFlower for a pretty basic task, the results were pretty bad.

Seems like with minimal oversight the human workers like to just say they did the requested task and make up an answer rather than actually do it (The task involved entering an address in Google maps, looking at the street view and confirming insofar as possible if a given business actually resided at the address in question, nothing complicated)

Edit: woops, mixed in the query with another reply that mentioned the human element XD

It seems only fair that the humans charged with doing the grunt work to build an automated fabulist would just make stuff up for training data.

Tit for tat and all that.

https://github.com/bigscience-workshop/petals seems to have some capabilities in that area, at least for fine-tuning.

folding@home has been doing cool stuff for ages now. There's nothing to say that distributed computing couldn't also be used for this kind of stuff, albeit a bit slower and fragmented than running on a huge clusters of H100 with NVLink.

In terms of training feedback I suppose there's a few different ways of doing it. Gamification, mech turk, etc. Hell free filesharing sites could get on the action and have you complete an evaluation of a model response instead of watching an ad

Hate to say it, but this sounds like a legitimate use case for a cryptocurrency.

Check out Open Assistant for the reinforcement side of that dream.