Many companies overinvest in fully-owned hardware, rather than renting from clouds. Owning hardware means you underwrite unrented inventory costs and prevents you from scaling. H100 pricing is now lower than any self-hosted option, even without factoring the TCO & headcount.
(Disclaimer: I work at a GPU cloud Voltage Park -- with 24k H100s as low as $2.25/hr [0] -- but Fly.io is not the only one I've noticed purchase hardware when renting might have saved some $$$)
jonathan [at] tensordock.com
I'm curious what some of the numbers mean: e.g. what does 688/1464 GPUs available indicate in the left gray box? What about there being 1040 GPUs in light gray, and 8 in dark gray?
But even looking at the Lambda Labs benchmarks, I am surprised that the H100 PCIE barely outperforms the A100 SXM, for example. And it is meant to be a replacement for the A100 PCIE. 20% generational improvement yes, but I would have expected more?
I was playing around with some different GPUs yesterday and put all of the results here: https://www.tensordock.com/benchmarksI tried a vLLM and Resnet training workload. The H100 outperforms the A100 about 45% to 80% consistently, but it isn’t that much faster…What workloads would see the most speedup, because I’m really not seeing 3x+ any on vLLM or simple training workloads?
We do have a managed container hosting service, but it's built on our own backend that auto-scales nodes for you when average GPU utilization surpasses a certain point, but it's not K8s -- which would have been a pain to configure given the distributed nature of all of our servers.
https://dashboard.tensordock.com/deploy
Hey folks! I’m Jonathan from TensorDock, and we’re building a cloud GPU marketplace. We want to make GPUs truly affordable and accessible.I once started a web hosting service on self-hosted servers in middle school. But building servers isn’t the same as selling cloud. There’s a lot of open source software to manage your homelab for side projects, but there isn’t anything to commercialize...
We have our own supply base (sourced through https://tensordock.com/host), operate some of our own servers, and are not related to any other marketplace :)
We think we have better security & reliability than Vast.ai due to virtualization rather than Dockerization, as well as more strict access controls [1]. Additionally, you can run Windows VMs on us if you want :)
The IB equipped on our clusters (can't speak to others) is 8x 400 Gbps. Most customers training foundational models are able to fully utilize that fabric in parallel.
Instead, maybe consider trying to grow it on your own, see where you can take it, and if it shows some reasonable traction, a big player would probably definitely offer to swallow you up at a good offer so that they get the better technology that's already been demonstrated to work in a production environment.
Just my $0.02 :)
Happy to give you whatever credits you need (would $25 be enough?) to try running some cool stuff on our infrastructure. And feel free to email me the results of whatever you do at [email protected] :)
Even with $5 credits, we had people abusing the system and creating multiple accounts to cryptocurrency mine. So we chose $1 as an amount that would be so low that nobody would want to take advantage of, haha :)
We have data protection agreements with all of them, and we can also do bare metal machines on request so that you have full control over your physical machine.
It was started by Mark... he raises bees in his backyard, so we have some kick-ass honey raised right from our New England backyards :)
1 pound - https://buy.stripe.com/aEU28j7CR6tV81i7su 3 pounds - https://buy.stripe.com/8wM6oz3mB7xZbduaEH
We are actually working on something very similar to vast.ai (https://www.tensordock.com/product-marketplace) set to launch into a soft beta mode within the next two weeks and probably a real "Show HN" by the end of August. We'll have a few dozen GPUs scheduled to come online during the launch week at prices similar to Vast.ai. This would be with with full virtualization, which we think is better than Docker containers because we customers can run Windows VMs and do cloud gaming/rendering, thereby generating hosts more income. We might also add VM disk encryption later on, which would be more secure. Still, they are very large, so it'd be large road uphill, but we're working on something similar.
Also, if I remember correctly, with Vast (as a former user myself), an issue can arise when you have a VM in the stored state but someone is claiming the GPUs running an on-demand workload, which prevents you from being able to pull your data out. Because VMs are all booting from network storage and can be rescheduled to other compute nodes, you won't face that issue on our core cloud product here :)
- Jonathan @ TensorDock