News for hackers. Stuff that matters.
Re:GPT-3 is no longer the only game in town by DeathArrow on Sunday November 07, 2021 @11:39AM (#29140930)
People were blaming cryptocurrencies miners for the prices of GPUs, when in fact it was the AI researchers who bought all the GPUs. :D

I wonder what if somebody designs an electronic currency rewarded as payment for general GPU computations instead of just computing hashes? You pay some $, to train your model and the miner gets some coins.

Every one is happy, electricity is not wasted and the GPUs gets used for a reasonable purpose.

Re:GPT-3 is no longer the only game in town by PeterisP on Sunday November 07, 2021 @11:49AM (#29141052)
The current way of training is efficient when compute is located in a single place and is colocated with large quantities of training data. Distributing small parts of computation to remote computers is theoretically possible (and an active direction of research) but currently not preferable nor widely used; you really need very high bandwidth between all the nodes to constantly synchronize the hundreds-of-gigabytes sized weights they're iterating on and the resulting gradients.
Re:GPT-3 is no longer the only game in town by bckr on Sunday November 07, 2021 @01:08PM (#29141864)
This may not be true in the future. There is some work being done on distributed neural net training. I can't recall the name of the technique at the moment, but a paper came out in the last year showing results comparable with backprop that only required local communication of information (whatever this technique's alternative to gradients is).
Re:GPT-3 is no longer the only game in town by Nextgrid on Sunday November 07, 2021 @02:02PM (#29142398)
My understanding is that proof-of-work is intentionally wasteful; the objective is to make 51% attacks (where a single entity controls at least 51% of the global hashrate) infeasible by attaching a cost to the mining process.

Making the mining process produce useful output that can be resold nullifies the purpose as it means an attacker can now mine "for free" as a byproduct of doing general-purpose computations (as opposed to tying up dedicated hardware), lowering the barrier for a 51% attack dramatically.

Re:GPT-3 is no longer the only game in town by nathias on Sunday November 07, 2021 @11:46AM (#29141003)
Yes, this is an old idea (which I really like) but it hasn't really taken off yet. GridCoin was one example, where you solved BOINC problems or RLC that's for more general computation.
Re:GPT-3 is no longer the only game in town by rewq4321 on Sunday November 07, 2021 @11:50AM (#29141057)
The problem is that, currently, large ML models need to be trained on clusters of tightly-connected GPUs/accelerators. So it's kinda useless having a bunch of GPUs spread all over the world with huge latency and low bandwidth between them. That may change though - there are people working on it: https://github.com/learning-at-home/hivemind
Re:GPT-3 is no longer the only game in town by Kiro on Sunday November 07, 2021 @11:51AM (#29141074)
It hasn't taken off because it doesn't work. PoW only works for things that are hard to calculate but easy to verify. Any meaningful result is equally hard to verify.
Re:GPT-3 is no longer the only game in town by petters on Sunday November 07, 2021 @12:30PM (#29141474)
> Any meaningful result is equally hard to verify.

This is very much not true. A central class in complexity is NP whose problems are hard to answer but easy to verify if the answer is yes.

E.g. is there a path visiting all nodes in this graph of length less than 243000? Hard to answer but easy to check any proposed answer.

Re:GPT-3 is no longer the only game in town by snovv_crash on Sunday November 07, 2021 @02:09PM (#29142454)
It's easy to verify ML training - inference on a test set has lower error than it did before.

Training NN ML is much slower than inference (1000x at least) because it has to calculate all of the gradients.

Re:GPT-3 is no longer the only game in town by technerder on Sunday November 07, 2021 @07:42PM (#29145078)
It'd be awesome if instead of the proof of work algorithms being based on generating useless hashes, they were instead based on computing protein folding simulations in a way that actually benefits everyone.
Re:GPT-3 is no longer the only game in town by magikabula on Sunday November 07, 2021 @05:16PM (#29144042)
If everyone offer GPUs, is the same game. If I will buy more GPU I will get more money, so the average payment for a person with a single or a small bunch of GPU will be low.

And second, the principles of electronic currency are different from gold/money. That's why crypto uses GPU ;)