I know someone who has had multiple 3090s for a couple years now, using them exclusively for ML(he's a data scientist) and none of them have failed. So I'm not sure which models you have in mind when you say "consumer grade" - the 3090(and the 4090 by extension) were always meant as replacement for the Titan series, which wasn't exactly "consumer" level.

Ah did not know that. I thought they were marketed towards consumers.

I never had one fail, but I had to go through some big hurdles to use them at work (University research lab) properly.

Namely, there are 2 big issues: - Virtualization: Getting vGPU functionality out of them (most of the time, there's no point in having a full 1080 Ti for only 1 job, we want to have it split on more than 1 VM). - Kubernetes workloads: I'm running a simple workload that uses the GPU (for testing for instance) and k8s wants it all for only one pod.

Just set the required/requested gpu number to 0 and k8s will let you use same gpu with multiple pods.

You'll need to make sure to not overload gpu mem, otherwise you'll get oom errors. You can use k8s patch functionality to add custom resources, essentially use that to represent gpu mem and include the amount of gpu the pod uses in its definition, that way you don't go over the limit.

I have a bunch of 3090s in the garage for machine learning almost 24x7. The garage has no AC at all. None of them have failed yet.

What have your machines learned so far?

Whether or not something is a hotdog, and the most optimal way to order ground beef.