saturn99

I think FPGAs would be an awesome prototype but maybe too constricting in terms of resources? The extrapolation might be so far out to be just as accurate as their simulated model...

Seems to align with Google, seems like they release a new TPU every 1.5 to 2 years.

I think modern LLMs are powerful enough now that they will still be useful in a couple years even if they aren't state-of-the-art. ChatGPT still lets you run their older model for cheaper than running GPT4, I could see a world where GPT4 is still available in 1.5 years even if there are better models out there.

Yep, pretty unique system. I imagine anything that runs LLMs better than we currently can would pique the interest of VCs.

I think the costs are from the Moonwalk model, which is a pretty good reference for estimating costs, although it might be low if you use all Google engineers to build the HW. =P