For me, the second-biggest obstacle would be data size and correlated, data transfer speed. I regularly analyse GBs to TBs of data on supercomputers which have PBs of free space, regular consumer PCs don't have the harddrive space for that kind of thing. And even if they have, they may not have the transfer speeds to make copying my data there feasible - by the time copying has finished, I may have already run the analysis on a slower CPU

I don't work with ML, so maybe this is stupid, but wouldn't a "regular consumer PC" have orders of magnitude less GPU power than a "supercomputer"? I'm assuming the latter has some form of GPU farm, whereas I don't know if you can fit more that 3-4 fat GPUs in the former.

So if you'd be able to split the workload such that it would be able to run on consumer PCs in a reasonable time, wouldn't that also split the storage requirements the same way?

Or, if instead you're OK with waiting for ages for the regular GPU to do its thing, is transfer speed that much of an issue?

Yes there are models that are trained on hundreds of GPUs but from my limited experience in scientific computing, most of the time researchers run their programs on a single node because going multi-gpu or multi-cpu requires somewhat large code changes and a single node is "good enough" for their use case or they come from a heavy science background and don't even know how to utilize multi node architecture. Their main benefit from using a cluster is 100% uptime, large storage, and large memory. I've been to multiple research institutes where there is an institute-wide HPC and researchers share it, that way no one needs any kind of high end computer and can just connect to the cluster. This service can help in that area if researchers can somehow schedule a job from their low-end laptop and get the results when the job is done.

Are the PBs of free space necessary? I thought training a ML model on a terabyte of data was computation expensive, but not storage intensive.

Sometimes! Not for the GPU part, but to make the summary data for the ML data.

For example, genome sequencing data and intermediate results are easily in the TB area of space, but the resulting table of genomic variants (k-mers like n-grams in NLP) is only a few hundred GB.

I'm confused from your post where the petabytes of storage come in.

I'm guessing that there are just many data sets, and each one is in the TBs of size.

> The founders described it as "kinda like airbnb for compute" here:

"... (business model: Uber for GPU waste heat)"

I immediately thought of n-gate and then laughed out loud at this. I miss n-gate.

so an open market for scalpers to sell you a "GPU time share"...

Sorry for the tangent, but I followed the thread, and something that confused me was this:

> Power outages, well there's not much you can do about that, again I don't know where you live, but those are pretty rare where I am. Maybe once a year for a few hours.

Once a year, a few hours. That’s not pretty rare, that’s close to frequent. I’ve had two outages in 17 years, only one of those was longer than a few minutes.

99.95% uptime is a few hours a year of downtime. Some people lose power almost daily. The thing about exponential / logarithmic distributions is your sense of what is frequent or rare is quite arbitrary and based on your own experiences which are rarely all that representative.

When I lived on country roads miles out of town power outages happened many times a year, occasionally for days. Some I’ve lived in big buildings with buried power lines I’ve never had an outage in years. People in developing economies in some places only have occasional electricity. Is quite hard to say which one is normal.

We have 5-10 outages a year where I live (rapidly growing city in the US), usually 1-3 hours in length but occasionally longer.

The power lines are all above ground and there are lots of trees. Every big storm brings a substantial risk of knocking the power out.

Wow, I did not know that. For me, outages are connected to major disasters.