Time for a dedicated "AI box" at home with hotswapping compute boards? Maybe put it inside a humanoid or animal-like robot with TTS capabilities?

Sign me up for that kickstarter!

each billion parameters using 32 bit floats requires around 2 GB of GPU or TPU RAM. ChatGPT is expected to have around 1000 billion. Good open source LLMs have around 7-20 billion currently. Consumer GPUs currently max out at 24 GB. You can now quantize the model to e.g. 4 bits instead of 32 per parameter and do other compressions, but still there is quite a limit what you can do with 24 GB of RAM. The Apple unified memory approach may be a path forward to increase that... so one box gives you access to the small models, for a GPT4 like model you'd need (for inference and if you had the model and tools) probably 100 of those 4090s or 25 of H100 with 96 GBs I guess to fit in 2 TB of model data.

Currently we do not explore sparsity. The next iteration of models will be much more compact by focusing on reducing effective tensor size.

It seems like a ton of engineering effort has been put into these neural network frameworks. How didn’t they explore sparsity yet? With numerical linear algebra that’s, like, the most obvious thing to do (which is to say, you probably know beforehand if your problem can be mapped to sparse matrices).

From what I’ve read recently, most sparse methods just haven’t given that much improvement yet, and we’re only recently pushing up against the limits of the “just buy more RAM” approach.

It sounds like there is a lot of work happening on sparse networks now, so it’ll be interesting to see how this changes in the near future.

It's not at all strange to get something to work before you start optimizing. I mean if you can only run small models then how would you even know what you're losing by optimizing for space? Heck you wouldn't even know how the model behaves, so you won't know where to start shaving away.

I'm not saying it's impossible but if resources allow it makes a lot of sense to start with the biggest model you can still train. Especially since for whatever reason things seem to get a lot easier if you simply throw more computing power at it (kind of like how no matter how advanced your caching algorithm it's not going to be more than 2 times faster than the simplest LRU algorithm with double the amount of cache).

Sounds a bit like premature optimization ( to have done it _by_ now ) , I bet it's in the works now though.

"Do things that don't scale" – PG

Gpus were built for dense math and they ran with it. To the point current best architectures are in part just the ones that run best using the subset of linear algebra gpus are really good at.

There has been a lot of work on sparsity and discovering sparse subnetworks in trained dense networks. And intel even proposed some alternative cpu friendly architectures and torch/tf and gpus are starting to do okay with sparse matrixes so thing are changing.

It is quite likely GPT-4 uses one or even two sparsity approaches on top of each other (namely, coarse grained switch transformer-like and fine grained intra-tensor block sparsity), if you look at the openly available contributors' research CVs.

Google, in collaboration with OpenAI, has published an impressive tour de force where they have throughly developed and validated at scale a sparse transformer architecture, applied to general language modeling task: https://arxiv.org/abs/2111.12763

This happened in November of 2021, and there is a public implementation of this architecture on the Google's public github.

Impressively, due to some reasons, other up-and-coming players are still not releasing models trained with this approach, even though it promises multiplicative payoff in inference economy. One boring explanation is conservatism for NN training at scale, where training runs cost O(yearly salary).

Let's hope the open source side of things catches up.

not in collaboration with openai, one of the authors joined openai before the paper was written and arxiv'd

At least from my experience with sparse matrix libraries like Eigen, you need to get the sparsity down to about 5% before switching from a dense to a sparse algorithm gets you execution time benefits.

Of course from a memory bandwidth and model size perspective maybe there are benefits long before that.

I wonder if an outdated chip architecture with just a lot of (64GB?) GDDR4 or something would work? Recycle all the previous generation cards to super high VRAM units

I've seen reports of people being able to run LLMs at decent speeds on old Nvidia P40s. These are 24GB Pascal GPUs and can be bought for as low as $100 (although more commonly $200) on eBay.

Link to report please

You can do it on CPU now.

Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34

It should work. After the initial prompt processing, token generation is typically limited by memory bandwidth more than by compute.

Intel Larrabee has come into the chat but showed up about decade and a half too early.

llama-65b on a 4-bit quantize sizes down to about 39GB - you can run that on a 48GB A6000 (~$4.5K) or on 2 x 24GB 3090s (~$1500 used). llama-30b (33b really but who's counting) quantizes down to 19GB (17GB w/ some optimization), so that'll fit comfortably on a 24GB GPU.

A 4-bit quantize of a 1000B model should be <600GB, so would fit on a regular 8x80 DGX system.

For completeness, Apple’s consumer GPUs currently max out at 64 GB - OS overhead, so about 56 GB. But you are limited to 1 GPU per system.

Benchmarks for what you can do on CPU alone.

https://github.com/ggerganov/llama.cpp/issues/34

An M1 Max does 100ms per token. A 64 core threadripper about 33ms per token.

It's more likely that you want a lot of compute for a very little amount of time each day - which makes centralised/cloud processing the most obvious answer.

If I want a response within 100ms, and have 1000 AI-queries per day, that would only be about 2 minutes of aggregated processing time for your AI box per day. It's less than 1% utilised. If the same box is multiuser and on the internet, it can probably serve 50-100 peoples queries concurrently.

The converse is that if you put something onto the cloud, for the same cost you might be able to effectively get 50x the hardware per user for the same cost (i.e. rather than have 1 AI box locally with 1 GPU for each of the 50 users, you could have 1 AI box with 50 GPU's which is usable by all 50 users).

"a lot of compute for a very little amount of time each day" sounds like something I can play games on when I'm not working.

Why not just buy a computer that is correctly-sized to play games, rather than buy an AI-sized computer that you mostly use for games?

Because I want both.

But not use both at once?

Well... No. If I'm sat playing a game, I'm unlikely to be generating AI queries.

The article talks about this explicitly though. Reasonably good models are running on raspberry Pis now.

Is a reasonably good model what people get value out of though?

Maybe this is why Sam Altman talked about "the end of the large LLMs is here"? He understands anything bigger than ChatGPT-4 isn't viable to run at scale and be profitable?

Does this mean models larger then ChatGpt would still be better for the same data size as long as someone is ready to pay?

At what limit does it stop getting better?

I thought he was fairly explicit that he thought larger models would provide incremental gains for exponentially greater cost, so yeah, I guess not profitable is a way to put it...

It's a win for Google that LLMs are getting cheaper to run. OpenAI's service is too expensive to be ad-funded. Google needs a technology that's cheaper to provide to maintain their ad-supported business model.

Google could make a bet like they did with YouTube.

At the time, operating YouTube was eye wateringly expensive and lost billions. But google could see where things were going: a triple trend of falling storage costs, falling bandwidth and transmission costs (I’m trying to dig up a link I read years ago about this but google search has gotten so shit that I can’t find it).

It was similar for Asic miners for Bitcoin. Given enough demand, specialised, lower cost hardware specially for LLMs will emerge.

On the flip side, I found only one person (I'm sure there are more) that are attacking the software efficiency side of things. You would be quite surprised how inefficient the current LLM software stack is, as I learned on a CPP podcast [0]. Ashot Vardanian has a great github repo [1] that demonstrates many ways compute can come way down in complexity and thus cost.

[0] https://cppcast.com/ai_infrastructure/ [1] https://github.com/orgs/unum-cloud/repositories?type=all

You realize that most reports are that YouTube is barely profitable.

This cost argument is being overblown. While it's a limitation for today's product, enginners are very good at optimization. Therefore the costs will drop in the medium to long term from efforts on both the software and hardware side.

Within a decade mid-level consumer cards will be just as powerful as those $40k cards.

Considering how long it took mid level consumer cards to beat my $600 1080, you're way more optimistic than I am.

The main moat here is VRAM, not raw compute power.

So... Four years? The 1080 launched in 2016, and the 3070 launched in 2020, for $100 cheaper — the launch price of the 1080 was $699, and the 3070 was $599. The 3070 easily trounced the 1080 in benchmarks.

The 3060 effectively matched a 1080 at $329 in 2021 (and has 50% more VRAM at 12GB instead of the 1080's 8GB), so call it five years if the 3070 isn't mid-range enough.

The 3060 Ti launched in 2022 at $399 and handily beat the 1080 on benchmarks, so call it six years if you want the midrange card to beat (not just match) the previous top-of-the-line card, and if a *70 card doesn't count as midrange enough. Less than a decade still seems like a reasonable claim for a midrange card to beat a top-of-the-line card.

are you conveniently forgetting how none of those cards were actually available for consumers to buy

The 1080 was impossible to buy at launch too and was sold out for months. And the 3060 is easy to buy!

The 3060 is easy to buy NOW, since cryptomining has crashed, and the 40x0 GPUs have become available (though mostly still above MSRP).

The 3060 was only readily available quite recently, so it's about 6 years from ready availability of the 1080 at $600 to 3060.

Taking 6 years to double the perf/$ implies that it would take ~42 years for a $40000 H100 to reach mid-range levels. Assuming scaling, particularly VRAM, holds.

And plus it would be getting really close to the Landauer limit by that point.

Given how nvidia has almost no competition, it just seems unlikely that nvidia decides to stop milking the enterprise and they will continue to lock 40GB+ cards behind ludicrous price points

Nvidia may not need to price compete immediately, but if LLMs drive demand for more capable consumer hardware they can either:

(1) make a whole lot of money fulfilling that demand, or

(2) leave an unmet demand that makes it more attractive for someone else to spend the money to field a solution.

(1) seems an attractive choice even if it wasn’t for the potential threat of (2).

>but if LLMs drive demand for more capable consumer hardware they can either:

That's a big if. I imagine people who will buy GPUs to run LLMs locally are either researchers or programmers. I am imagine most consumer focused solutions will be cloud first. That is a much smaller market than gamers and nvidia wouldn't want to cannibalize their datacenter offerings by releasing something cheaper. It's far better for them to sell high tier GPUs to Amazon and let them "rent" it out to researchers and programmers.

It wasn't ML but gaming that drove the demand for GPUs, and ML sort of rode in the slipstream (same for crypto). Later demand for crypto hashing drove GPU sales as much as gaming but that is now over. So unless either (1) ML by itself can present a demand as large as either crypto or gaming or (2) Crypto or gaming can provide a similar demand as they've done in the past the economies of scale that drove this will likely not be reached again. If they do however the cost of compute will come down drastically and that in turn may well drive another round of advances in models for the larger players.

You can run it (quantified at least) on a $4000 Mac thanks to Apple's unified memory. Surely other manufacturers are looking for how to expand VRAM, hopefully Intel or AMD.

Not to mention Apple chips have a bunch of very nice accelerators and also (!!!) macOS contains system frameworks that actually use them.

You're right and this is why they didn't heavily use BERT(in the full sense), arguably the game-changing NLP model of the 10s. They couldn't justify bringing the cost per search up.

nah, Lora quantized LLM’s are going to be at the OS level in 2 years and consumer architecture refreshes are just going to extend more RAM to already existing dedicated chips like Neural Engine

client side tokens per second will be through the roof and the models will be smaller

LoRa is not a quantization method, it's a fine tuning method.

you read that whole paragraph and assumed this prediction didn't involve consumers using fine tuned models despite lora being explicitly mentioned?

-EQ moment

edit: its about the combination of those methods making models accessible on consumer hardware

It's two adjectives on the noun.

The biggest llama model has near 100% fidelity (its like 99.3%) at 4 bit quantization, which allows it to fit on any 40GB or 48GB GPU, which you can get for $3500.

Or at about a 10x speed reduction you can run it on 128 GB of RAM for only around $250.

The story is not anywhere near as bleak as you paint.

I haven't seen any repos or guides to using llama on that level of RAM, which is something I do have. any pointers?

Run text-generation-webui with llama.cpp: https://github.com/oobabooga/text-generation-webui

And here are the benchmarks

https://github.com/ggerganov/llama.cpp/issues/34

Something I haven't figured out: should I think about these memory requirements as comparable to the baseline memory an app uses, or like per-request overhead? If I needed to process 10 prompts at once, do I need 10x those memory figures?

It’s like a database, I imagine - so the answer is probably “unlikely,” that you need memory per-request but instead that you run out of cores to handle requests?

You need to load the data so the graphics cards - where the compute is - can use it to answer queries. But you don’t need a separate copy of the data for each GPU core, and though slower, cards can share RAM. And yet even with parallel cores, your server can only answer or process so many queries at a time before it runs out of compute resources. Each query isn’t instant either given how the GPT4 answers stream in real-time yet still take a minute or so. Plus the way the cores work, it likely takes more than one core to answer a given question, likely hundreds of cores computing probabilities in parallel or something.

I don’t actually know any of the details myself, but I did do some CUDA programming back in the day. The expensive part is often because the GPU doesn’t share memory with the CPU, and to get any value at all from the GPU to process data at speed you have to transfer all the data to GPU RAM before doing anything with the GPU cores…

Things probably change quite a bit with a system on a chip design, where memory and CPU/GPU cores are closer, of course. The slow part for basic replacement of CPU with GPU always seemed to be transferring data to the GPU, hence why some have suggested the GPU be embedded directly on the motherboard, replacing it, and just put the CPU and USB on the graphics card directly.

Come to think of it, an easier answer is how much work can you do in parallel on your laptop before you need another computer to scale the workload? It’s probably like that. It’s likely that requests take different amounts of computation - some words might be easier to compute than others, maybe data is local and faster to access or the probability is 100% or something. I bet it’s been easier to use the cloud to toss more machines at the problem than to work on how it might scale more efficiently too.

Does that mean an iGPU would be better than a dGPU? A beefier version than those of today though.

Sort of. The problem with most integrated GPUs is that they don’t have as many dedicated processing cores and the RAM, shared with the system, is often slower than on dedicated graphics cards. Also… with the exception of system on a chip designs, traditional integrated graphics reserved a chunk of memory for graphics use and still had to copy to/from it. I believe with newer system-on-a-chip designs we’ve seen graphics APIs e.g. on macOS that can work with data in a zero-copy fashion. But the trade off between fewer, larger system integrated graphics cores vs the many hundreds or thousands or tens of thousands of graphics cores, well, lots of cores tends to scale better than fewer. So there’s a limit to how far two dozen beefy cores can take you vs tens of thousands of dedicated tiny gfx cores.

The theoretical best approach would be to integrate lots of GPU cores on the motherboard alongside very fast memory/storage combos such as Octane, but reality is very different because we also want portable, replaceable parts and need to worry about silly things like cooling trade offs between placing things closer for data efficiency vs keeping things spaced apart enough so the metal doesn’t melt from the power demands in such a small space. And whenever someone says “this is the best graphics card,” someone inevitably comes up with a newer arrangement of transistors that is even faster.

You need roughly (model size + (n * (prompt + generated text)) where n. Is the number of parallel users/ request.

It should be noted that that last part has a pretty large factor to it that also scales with model size, because to run transformers efficiently you cache some of the intermediate activations from the attention block.

The factor is basically 2 * number of layers * number of embeddings values (e.g. fp16) that are stored per token.

A $3500 GPU requirement is far from democratization of AI.

For reference, a basic office computer in the 1980s cost upwards of $8000. If you factor in inflation, a $3500 GPU for cutting tech is a steal.

Yes but moore's law ain't what it used to be

You're right, it's been debunked and misquoted for decades.

This is very true, however there is a long way to go in terms of chip design specific to DL architectures. I’m sure we’ll see lots of players release chips that are an order of magnitude more efficient for certain model types, but still fabricated on the same process node.

Moore is not helping here. Software and algorithms will fix this up, which is already happening at a frightening rate. Not too long ago, like months, we were still debating if it was ever even possible to run LLMs locally.

There is going to be a computational complexity floor on where this can go, just from a Kolmogorov complexity argument. Very hard to tell how far away the floor is exactly but things are going so fast now I suspect we'll see diminishing returns in a few months as we asymptote towards some sort of efficiency boundary and the easy wins all get hoovered up.

Yes indeed and it’ll be interesting to see where that line is.

I still think there is a lot to be gained from just properly and efficiently composing the parts we already have (like how the community handled stable diffusion) and exposing them in an accessible manner. I think that’ll take years even if the low hanging algorithm fruits start thinning out.

We've reached an inflection point, the new version would be: Nvidia can sell twice as many transitors for twice the price every 18 months.

Moore's law isn't dead. Only Dennard's law. See slide 13 here[0] (2021). Moore's law stated that the number of transistors per area will double every n months. That's still happening. Besides, neither Moore's law nor Dennard scaling are even the most critical scaling law to be concerned about...

...that's probably Koomey's law[1][3], which looks well on track to hold for the rest of our careers. But eventually as computing approaches the Landauer limit[2] it must asymptotically level off as well. Probably starting around year 2050. Then we'll need to actually start "doing more with less" and minimizing the number of computations done for specific tasks. That will begin a very very productive time for custom silicon that is very task-specialized and low-level algorithmic optimization.

[0] Shows that Moore's law (green line) is expected to start leveling off soon, but it has not yet slowed down. It also shows Koomey's law (orange line) holding indefinitely. Fun fact, if Koomey's law holds, we'll have exaflop power in <20W in about 20 years. That's equivalent to a whole OpenAI/DeepMind-worth of power in every smartphone.

The neural engine in the A16 bionic on the latest iPhones can perform 17 TOPS. The A100 is about 1250 TOPS. Both these performance metrics are very subject to how you measure them, and I'm absolutely not sure I'm comparing apples to bananas properly. However, we'd expect the iPhone has reached its maximum thermal load. So without increasing power use, it should match the A100 in about 6 to 7 doublings, which would be about 11 years. In 20 years the iPhone would be expected to reach the performance of approximately 1000 A100's.

At which point anyone will be able to train a GPT-4 in their pocket in a matter of days.

There's some argument to be made that Koomey himself declared in 2016 that his law was dead[4], but that was during a particularly "slump-y" era of semiconductor manufacturing. IMHO, the 2016 analysis misses the A11 Bionic through A16 Bionic and M1 and M2 processors -- which instantly blew way past their competitors, breaking the temporary slump around 2016 and reverting us back to the mean slope. Mainly note that now they're analyzing only "supercomputers" and honestly that arena has changed, where quite a bit of the HPC work has moved to the cloud [e.g. Graviton] (not all of it, but a lot), and I don't think they're analyzing TPU pods, which also probably have far better TOPS/watt than traditional supercomputers like the ones on top500.org.

0: (Slide 13) https://www.sec.gov/Archives/edgar/data/937966/0001193125212...

1: "The constant rate of doubling of the number of computations per joule of energy dissipated" https://en.wikipedia.org/wiki/Koomey%27s_law

2: "The thermodynamic limit for the minimum amount of energy theoretically necessary to perform an irreversible single-bit operation." https://en.wikipedia.org/wiki/Landauer%27s_principle

3: https://www.koomey.com/post/14466436072

4: https://www.koomey.com/post/153838038643

And hardly anyone had them in the 1980s.

virtually no office had them in 1980

by mid 1980s personal computers costed less than $500

This is false. Created account just to rebut.

While it may have been true that it was technically possible to assemble a PC for $500... good luck. In the real world people were spending $1500-$2500+ for PCs, and that price point held remarkably constant. By the time you were done buying a monitor, external drives, printer etc $3000+ was likely.

https://en.m.wikipedia.org/wiki/IBM_Personal_Computer#:~:tex....

Or see Apple Mac 512 introduced at approx $2800? One reason this was interesting (if you could afford it) was the physical integration and elimination of "PC" cable spaghetti.

https://en.m.wikipedia.org/wiki/Macintosh_512K

But again having worked my way thru college with an 8 MB external hard drive... which was a huge improvement over having to swap and preload floppies in twin external disk drives just to play stupid (awesome) early video games, all of this stuff cost a lot more than youre saying. And continued to well into the 90s.

Of course there are examples of computers that cost less. I got a TI-99/4a for Christmas which cost my uncle about $500-600. But then you needed a TV to hook it up to, and a bunch of tapes too. And unless you were a nerd and wanted to program, it didn't really DO anything. I spent months manually recreating arcade video games for myself on that. Good times. Conversely, if you bought an IBM or Apple computer, odds were you were also going to spend another $1000 or more buying shrinkwrap software to run on it. Rather than writing your own.

Source: I remember.

> This is false.

The CPC 464 is the first personal home computer built by Amstrad in 1984. It was one of the bestselling and best produced microcomputers, with more than 2 million units sold in Europe.

Price

£199 (with green monitor), £299 (with colour monitor)

> But again having worked my way thru college with an 8 MB external hard drive

that was a minicomputer at the time, not a PC (personal computer)

> Source: I remember.

Source: I owned, used and programmed PCs in the 80s

The Commodore 64 launched in 1982 for 595 bucks. Wikipedia says "During 1983, however, a trickle of software turned into a flood and sales began rapidly climbing, especially with price cuts from $595 to just $300 (equivalent to $1600 to $800 in 2021)."

I believe this was the PC of the ordinary person (In the "personal computer" sense of the word.)

I think a more relevant comparison may be a peripheral: the $7,000 LaserWriter which kicked off the desktop publishing revolution in 1985.

$3500 is less than one week of a developer salary for most companies. It woudn't pay a month's rent for most commercial office space.

It's a lower cost of entry than almost any other industry I can think of. A cargo van with a logo on it (for a delivery business or painting business, for example) would easily cost 10-20x as much.

* in the US

1. I don't know what kind of world you live in to think that USD 3500 is "less than one week of a developer salary for most companies." I think you really just mean FAANG (or whatever the current acronym is) or potentially SV / offices in cities with very high COL.

2. The problem is scaling. To support billions of search queries you would have to invest in a lot more than a single GPU. You also wouldn't only need a single van, but once you take scaling into account even at $3500 the GPUs will be much more expensive.

That said, costs will come down eventually. The question in my mind is whether OpenAI (who already has the hardware resources and backed by Microsoft funding to boot) will be able to dominate the market to the extent that Google can't make a comeback by the time they're able to scale.

1 - Yes, I agree on this, but even so, most developers already are investing in SOTA GPU's for other reason (so not as much of a barrier as purported)

2 - Scaling is not a problem in other industries? If you want to scale your food truck, you will need more food trucks, this doesn't seem to really do anything for your point.

GGML and GPTQ have already revolutionised the situation, and now there are tiny models with insane quality as well, that can run on a conventional CPU.

I don't think you have any idea what is happening around you, and this is not me being nasty, just go and take a look at how exponential this development is and you will realise that you need to get in on it before its too late.

You seem to be in a very particular bubble if you think most developers can trivially afford high end GPUs and are already investing in SOTA GPUs. I know a lot of devs from a wide spectrum of industries and regions and I can think of only one person who might be in your suggested demographic

Perhaps I should clarify, that when I say SOTA GPU, I mean, rtx 3060 (midrange), which has 12gb vram, and is a good starting point to climb into the LLM market. I have been playing with LLM's for months now, and for large periods of time had no access to GPU due to daily scheduled rolling blackouts in our country.

Even so, I am able to produce insane results locally with open source efforts on my RTX3060, and now I am starting to feel confident enough that I could take this to the next level by either using cloud (computerender.com for images) or something like vast.ai to run my inference (or even training if I spend more time learning). And if that goes well I will feel confident going to the next step, which is getting an actual SOTA GPU. But that will only happen once I have gained sufficient confidence that the investment will be worthwhile. Regardless, apologies for suggesting the RTX3060 is SOTA, but to me in a 3rd World Country, being able to run vicuna13b entirely on my 3060 with reasonable inference rates is revolutionary.

I'm only going to comment on the salary bit.

GP lives in an company world. The cost of a developer to a company is the developer's salary as stated in the contract, plus some taxes, health insurance, pension, whatever, plus the office rent for the developer's desk/office, plus the hardware used, plus a fraction of the cost of HR staff and offices, cleaning staff, lunch staff... it adds up. $3500 isn't a lot for a week.

Most of these items are paid for by the company, and most people would not consider the separate salary of the janitorial or HR staff to be part of their own salary.

I agree, most people wouldn't. This leads to a lot of misunderstandings, when some people think in terms of what they earn and others in terms of what the same people cost their employers.

So you get situations where someone names a number and someone else reacts by thinking it's horribly, unrealistically high: The former person thinks in employer terms, the latter in employee terms.

> 1. I don't know what kind of world you live in to think that USD 3500 is "less than one week of a developer salary for most companies." I think you really just mean FAANG (or whatever the current acronym is) or potentially SV / offices in cities with very high COL.

I live in the real world, at a small company with <100 employees, a thousand miles away from SV.

$3200 * 52 == $180k a year, and gives $120k salary and $60k for taxes, social security, insurance, and other benefits, which isn't nearly FAANG level.

Even if you cut it in half and say it's 2 weeks of dev salary, or 3 weeks after taxes, it's not unreasonable as a business expense. It's less than a single license for some CAD software.

> 2. The problem is scaling. To support billions of search queries you would have to invest in a lot more than a single GPU. You also wouldn't only need a single van, but once you take scaling into account even at $3500 the GPUs will be much more expensive.

Sure, but you don't start out with a fleet of vans, and you wouldn't start out with a "fleet" of GPUs. A smart business would start small and use their income to grow.

You are correct sir!

I thought this thread was about it becoming ubiquitous. If we're talking about just business use, that's a whole different thing.

and that 3500 worth of kit will be a couple of hundred bucks on ebay in 5 years.

Yeah, I bet people won't get cars, either, they're a lot more expensive than that.

That's just to run a model already trained by a multi-billion dollar company. And we are "lucky" a corporation gave it to the public. Training such a model requires tons of compute power and electricity.

> which you can get for $3500.

Which puts it out of reach for the majority of people, and means many people who could technically afford it will choose not to spend that much on it anyway.

I was in Japan recently and they sell these pocket size translator devices with a microphone, camera and screen. You can speak to it or take pictures and it will translate on the fly. Maybe $100 usd range for a nice one.

It's only a matter of time before someone makes a similar device with a decent LLM on it, and premium ones will have more memory/cpu power.

I think we as humans have a tendency to extrapolate from our present position to a position we can imagine that we’d like, even if there isn’t a foreseeable path from here to there. I believe this may end up being one of those cases.

Why? What GP describes seems both feasible and inevitable to me.

I mean...isn't that just a smartphone?

I know exactly what you're talking about because my father-in-law had the same thing. I'm just very skeptical that specialist hardware will overtake general commoditized computing devices for mass-market usage. The economics alone make it unlikely.

Yes, usually they are locked-down Android devices.

One of them has an unlimited "free forever" internet subscription so it can fetch network translations online if the local dictionary doesn't have the word.

Even though Word Lens was first released over 12 ago, I keep surprising people[0] by showing them the same technology built into Google Translate.

[0] even other tech developers who, like me, migrated somewhere where a language barrier came up

Yes, that is because Google bought the technology from Word Lens in 2015.

Wiki says: https://en.wikipedia.org/wiki/Ot%C3%A1vio_Good

    To develop Word Lens, Otávio Good founded Quest Visual Inc., which was acquired by Google, Inc. in 2014, leading to the incorporation of the Word Lens feature into the Google Translate app in 2015.

I think you misunderstand.

The people I show it to are surprised that it's possible in 2023, even though it was demoed at the end of 2010.

Or you can quantize the model and run it on your laptop.

Is the assumption that GPU power and advancements in AI will not get to a reasonable price point in the near future? Because it seems to me that advances in computation have not slowed down at all since it started.

Yeah, today.

Or you can apply GPU optimizations for such ML workloads. By optimizing the way these models run on GPUs, significantly improve efficiency and slash costs by a factor of 10 or even more. These techniques include kernel fusion, memory access optimization, and efficient use of GPU resources, which can lead to substantial improvements in both training and inference speed. This allows AI models to run on more affordable hardware and still deliver exceptional performance. For example, LLMs running on A100 can also run on 3090s with no change in accuracy and comparable inference latency.

Would google even care about integrating LLMs into search? They don’t even prune all the spam entries, presumably because they increase advertising revenue and analytics profit.

The thing you have in your pocket would have meant an enormous investment for equivalent compute power just decades ago and filled a whole basement with server racks.

The legendary Cray-2 was the fastest supercomputer in the world in 1985, with peak 1.9 GFLOPS. Less than four decades ago.

By comparison, the Cray is outperformed by my smartphone.

Actually, it is outperformed by my previous smartphone, which I purchased in 2016 and replaced in 2018.

Actually, it is outperformed by a single core on my previous smartphone, of which it has eight cores.

caching + simpler models for classification / triage should reduce the load on the big model.

We used to get only a limited number of Internet hours. By the time December 2003 rolled around my family had always on internet.

Besides what Google does here is besides the point, because Bing has already unleashed ai search. Google will either follow along or stop being relevant.