One of the perks of crypto losing value. GPUs finally being used for breakthrough research, rather than pyramid schemes.

This claim feels truthy, but is it true? Most of the big models were being trained on specialised hardware like tensor chips right?

And the biggest drop isn’t because crypto is worth less, it’s because Ethereum doesn’t use proof of work since November (Bitcoin hasn’t used GPUs for years). The explosion in novel AI models we saw last year predates this change.

Most big AI is trained on Nvidia GPUs but usually not the standard consumer ones found in the GeForce line-up. Instead it's usually their data centre GPUs like the current A100 or soon to be H100 that's just hitting the market.

Google does have their TPUs(Tensor Processing Units), however it is not cost efficient budget wise, so unless you have some kind of deal with Google or compute credits it doesn't make sense. They have pods upon pods of TPU clusters though so the main selling point of TPU training is that you can get your training done really fast with just the ease of scaling your workload to more TPUs.

So if you needed a big model like GPT-3 trained in a single day, you could spend an ungodly amounts of money and get it done with Google TPUs. Otherwise if you can wait weeks or months you can go with the standard Nvidia data centre solution and it'd be cheaper at the end by a significant margin.

Crypto has been running on ASICs for years. The "evil crypto miners make your GPUs so expensive" psyop is a NVIDIA campaign to rise GPU prices.

True, but that doesn't mean that crypto miners aren't still evil.

In 1993, it was said that the game of Go would be the drosophila fly of artificial intelligence. Alphago (then AlphaGo Zero) has succeeded in 2016, 2017. The glass ceiling has been broken. We are witnessing the birth of a new world where artificial intelligence will be more and more present in everyday life. I do not see any limit.

> I do not see any limit.

Hmmm, now I'm wondering if you're actually an advanced AI that has already simulated what's going to happen.

There have been some pretty big advances in the last few months in machine learning.

And Open AI's Whisper, which for me is the cherry on top!

Jesus it can even understand Scottish. I need this app!

Obligatory Burnistoun reference: https://www.youtube.com/watch?v=TqAu-DDlINs

But I'd be interested to see what its reliability is like on various types of Scottish accent.

I tried Whisper on an admittedly very challenging task (live translate Albanian to English subtitles), and it failed miserably. Went back and saw it had a 44% error rate for that language.

Might give it another go on an easier task.

Even though Whisper advertises the translation part of it. I honestly wouldn't say it's a first class feature at least part of the Whisper model itself.

The only thing I'd use Whisper for is transcription part of it. Then use ChatGPT for the translation.

Yeah the top 5 languages on that WER graph is very much worth a look, the rest a bit error prone.

Surprisingly, I tested it with my mother, she has a very broken english accent/dialect and it worked fine for her. Works amazing actually and I'm busy building some tools around it for her to test.

Breakthrough tech doing breakthrough tech things

It's the new hot topic?

The cynical view would be that in the current tech environment and generalized risk aversion rushing to demonstrate tangible use cases for AI will help justify a decade of out-of-control hype and investment, support exits, new funding runds etc.

Underneath all the commercial interest some specialists inevitably must keep adding to the knowledge base / capabilities but good luck finding any objective account of that process.

The paper about the transformer architecture was published in 2017 [1]. My understanding is that it took a few years, but we are now seeing the results of that breakthrough.

[1] https://arxiv.org/abs/1706.03762

Imagine Hacker News existed when Netscape Navigator was released. (Maybe it was a Usenet discussion board or something.) Suddenly everyone would be talking about the World Wide Web.

At the moment, ChatGPT and Stable Diffusion look like the same order of magnitude of advance.