Thanks so much for the detailed recommendation! We're thrilled that you're with us, that you appreciate our low prices, that you find our Cloud Console pleasant to use, and that our rescale feature helps you grow so easily! --Katie

Please do a managed Kubernetes next. I couldn't convince any of our customers to switch to Hetzner because they'd need to do "everything themselves". A managed Kubernetes instance would instantly make Hetzner an alternative for at least 75% of our customers. And honestly it's quite a cheap way to earn a bonus on your server instances.

Edit: And if you do manged Kubernetes and managed Kafka Instances the number would go up to like 95%. Oh and those Videos with der8auer? Really awesome to see, do Linus Tech Tipps or Level1Techs next!

Kubeone on Hetzner is honestly one of the smoothest non-managed experiences I've ever had with Kubernetes - better than some managed ones. Includes all of the things Hetzner already provides for Kubernetes (so I'm sure they're working on their own) - internal networking support, load balancers, volumes. And you scale worker nodes with a CRD too.

https://github.com/kubermatic/kubeone/tree/main/examples/ter...

That's wicked cool, thank you. For me and a little more experimental customers, this is a nice option but for the more legacy customers out there, if it doesn't have an SLA, it doesn't exist. I know it doesn't make sense but it's how big companies tend to work.

Oh, I've been there. I just wanted to highlight an outstanding option.

Though I'm sure if you ask and they have the money for it lying around, Kubermatic will sell you an SLA.

I don’t think Hetzner is the company to provide that. The reason they’re so cheap is that they limit themselves to hardware/VPS.

They do have managed offerings.

That product is at least one and a half decade old though. I wouldn't call any of their new products "managed".

Hey y'all -- I'm one of the core maintainers for KubeOne. I'm super happy to hear you had an awesome experience!

For folks wanting to learn more, we have this nice getting started tutorial covering all the cloud providers we support including Hetzner: https://docs.kubermatic.com/kubeone/v1.5/tutorials/creating-...

And for folks asking about SLAs, support, and stuff like that, I recommend checking out our KubeOne Managed Offering: https://www.kubermatic.com/products/managed-kubermatic-kuber...

I'm also happy to answer any questions y'all might have. :)

> Please do a managed Kubernetes next.

If we're taking requests, I would love a Fargate-like service. It would be absolutely fantastic if Hetzner provided a service where users uploaded a Docker image, and Hetzner took care of handling the networking bits and running it, and gather metrics.

Something like this is on my roadmap, would you mind telling me a bit about the metrics and the scale you'd expect? Would you expect always on or more of an ephemeral container?

AWS Fargate / GCP Cloud run

Upload a docker image, specify container size (1cpu 2gb)

go live

scale from 1rps to 1000 rps any time

stateless

pay per request or pay per container

Combining what the original comment was, they also want some AppRunner-style ergonomics -- I'd like to see just how much of cloudwatch/monitoring would be expected to be available. Basic things like CPU and memory aren't too hard but it really does depend on how much of the VM (firecracker/etc) one would expect to be able to see, as well as higher level metrics like (RPS/Errors, etc).

AppRunner, Fargate and Cloud Run have different ergonomics and specifics but thanks for this outline. v1 is likely to be pay per container but other than that feels doable.

> Something like this is on my roadmap,

Superb!

> would you mind telling me a bit about the metrics and the scale you'd expect?

It will depend mostly on what the service offers.

If the service only supports running a single isolated container without any scaling whatsoever then it would be helpful if we could monitor básico stuff like CPU and memory utilization, and also network traffic, free disk space, and also disk IO. If the service supports auto-scaling then it would be helpful to track all resource utilization rates along with all alarms and events involved. Auto-scaling also implies load balancing thus if that's the case then it would also be helpful to track the basic load balancing indicators, as well as request logs.

In the end it really depends on what services you're planning on offering, and how you'll charge for it. As a user I would need to monitor any metric which is directly and indirectly involved in determining cost, and on top of that I need to monitor performance.

> Would you expect always on or more of an ephemeral container?

The most pressing need would be always on containers to be able to go the lift-and-shift onboarding route to managed services, but ephemeral containers sound like function-as-a-service and those are pretty exciting as well.

Thanks for this incredibly detailed answer! All these points make a ton of sense.

Free disk space would imply elastic block storage or something similar so I’ll need to give that a think!

V1 is very likely to be always on so great that it’s the core use case for you!

They could do it like GCP or Scaleway - provide a managed Kubernetes service, and provide managed Knative on top for the Container as a Service service.

Managed k8s also requires stateful services like storage, database and ingress/LB which I recommended Hetzner to build like 8 years ago (community).

Proof: https://forum.hetzner.com/index.php?thread/21421-docker-cont...

Hetzner S3 Storage would be great. I know Hetzner is slow moving when it comes to Software but it's so obvious that they would have explosive growth if they did that and most of it is available as FOSS Software already

So I'm working on adding this on top of Hetzner -- it's service #2 and the beta should be out by the end of the month (Redis is first!) -- if you'd be willing to beta test I'd love to have you try it.

You can find the link in my profile or here[0]

[0]: https://nimbusws.com

IMHO this has to be offered by the platform, not by some other party/intermediary. Of course I could run minio, swift, etc by myself, but this is not the point.

Thanks for sharing this, hopefully after the expansion finishes (and maybe some dedicated servers appear in the US), Hetzner could offer object storage natively!

I have seen the "apps" in image selection in the Hetzner cloud console, so I'm convinced they're at least thinking about going up the value chain.

Hard disagree, this would be awesome. I'm leaving my current position but would have considered this as an unblocker for moving from Azure to Hetzner.

Would you have wanted a hypothetical discussion though (directed at grandparent), I would be up for that. Email in my profile

Thanks for voicing this! I would certainly welcome feedback, email incoming!

They would also have explosive costs and expenses.

There are some other pieces missing. The cluster-autoscaler implementation is quite basic. It doesn't have a way to managing multiple clusters inside the same project and no way to add custom tags. That makes it harder to track resources. The permissions/security is quite simple. Read/write tokens with full permissions on the project is very wide. Adding S3 like service without more security would be interesting. These are the most important ones, but I remember there were a few more nice to have feature (like server groups for scaling up an down and the ability to retrieve the userdata, image and ssh keys for a running server.

Never heard of Level1Tech, just looked them up. Man... These people are so likeable! Great content!

For Kubernetes, maybe give kOps a try. It's quite close to managed. The Hetzner integration is quite good (I spent about 6 months getting it ready for my company). The community support is great also. It still needs autoscaling added though.

If you need managed K8s and Kafka (and pretty much anything else you can self host), my company [1] provides just that with 5x16 real human support.

[1] https://www.ayedo.de

https://symbiosis.host uses Hetzner for their underlying nodes as I understand. It certainly explains them being the cheapest managed Kubernetes offering I've ever seen.

Are you going to be hiring in the US?

How are their APIs for Configuration as Code?

Pretty good.

https://registry.terraform.io/providers/hetznercloud/hcloud/...

I wonder when Aiven https://aiven.io/ (or something similar) will start supporting hetzner.

A few months ago I heard that Aiven was warming up to collaboration with OVH, which is even better IMO.

Whenever I hear Aiven, I think of an incident I witnessed 2nd hand in January 2020. They accidentally terminated the services for (at least) one big customer and had to restore them from backups. This lead to data loss in production (Kafka topic data and configuration gone) and was a huge mess for the customer to clean up. Of course they abandoned Aiven after that.

You might be interested by Elestio, we support 13 managed DB and 170 other open-source software. We also support hetzner including this new Hillsboro region.

https://elest.io/

Disclaimer: I'm the CTO

> Cloudflare R2 - another great value S3 alternative with $0 egress fees

I don't understand their price page. They claim $0 egress fees, but their free "Class A operations (mutate state)" and "Class B operations (read state)" have a mothly cap. After that you pay by the number. Isn't that an egress free?

No, because you pay by the operation, but by the amount of data in that operation. S3 also has operations costs, but those are separate from the egress costs, which are throb the roof.

Does anyone know what the actual margin on outbound data / egress is?

S3 is charging $0.023 per GB, right?

IIUC - there's not a fixed cost for sending 1 GB of data. The cost differs based on where you send from and send to. So it'd be hard to have a really good estimate - but I'm wondering if anyone has a good ballpark figure.

My understanding is that transferring data on the same continent (vast majority of traffic) should be <$0.0005 - meaning the margin is really high.

I know it's more in Amazon's interest for your cloud usage to be as inefficient as possible - so they can charge you as much as possible and get as much margin as possible.

However, this product doesn't seem to make sense to me.

Why would you even pick your S3 regions? Shouldn't AWS balance your data for you across continents so that your data egress is automatically optimized?

Is that how Cloudflare works?

That $0.023 per GB is for storage. It's $0.09 per GB for egress (varies by region). It's charged the same as all other egress from AWS.

Amazon egress changes depends on the region of the s3 data location.

Bandwith cost highly depends on the scale and target network. Cloudflare has a good blogpost on relative bandwith cost: https://blog.cloudflare.com/bandwidth-costs-around-the-world...

To put numbers to the relativity, for a somewhat smaller datacenter in Central Europe with a 100Gbit/s connection, the running Cost of transit is somewhere around 0.00004€ per GB, on a theoretically fully sustained connection(not realistic, also with no redundancy or hardware). Peering is basically free(after initial buildout) with around 3k€ per 100Gbit/s (https://www.ams-ix.net/ams/pricing). This all excludes cost of investment for initial buildout and hardware. On a scale on amazons level this becomes even cheaper, since operating your own network is cheaper than buying transit.

Hetzner charges 1,19€ per TB, amazon 90$(on the first 10 TB). So amazon probably doesnt really care about the price differences per region and went with a mixed calculation, since their margins are so absurdly huge.

also, major parts of the cost are in hardware and staffing/development, of which amazon has way more, due to the demands of SDN.

I run pixeldrain.com which uses a lot of bandwidth. The website uses about 6 PB per month and I'm paying €3000 for that, so that's about €0.50 per TB

Not if you download a small number of large files ;)

Adds up really fast!

One time I hosted a new movie (mkv, 23gb+), that cost like $17 to “rent” on Amazon prime… so my extended family could just take the link, download it, or just watch it in their mobile browser or laptop.

The egress of the streaming alone (not downloading), while they aircasted, definitely adds up much more than one would expect!

First thing on signup: YOU HAVE TO DO ADDITIONAL IDENTITY VALIDATION, yes, even though we took your home address and credit card and phone number already.

No thanks, I'll keep using Digital Ocean or someone who doesn't make me jump through hoops.

A one-off hoop that maybe helps to keep spammers off their systems ? Fine by me.

If your fraud detection cant do anything with name, address, phone number, and credit card and you need Drivers License/Passports then you probably need to pay for a better one.

What do you expect a German company to do with a copy of your drivers license or passport? Is this one of those situations where it contains your your social security number and it therefore becomes direct access to your bank account?

The only thing I imagine someone could do with the copy of my passport is pretend to be me when they sign up for a similar service.

Yes, pretend to be me and charge me for their actions. We call this identity theft and it is a rampant problem.

Interestingly, I've never used Digital Ocean because when I went to sign up (in my memory it would be circa 2015, but my memory is not what it once was) you had to give them something like your Twitter or Github name, which I didn't feel like sharing.

I signed up to DO about 2 years ago. They didn't ask me for Twitter or GitHub. As I recall, it was a normal setup process.

eBay did this to me. They let me enter everything and then said they can't validate my ID without any way to fix that even. It's a nice way to provoke people for sure.

Thanks for the notification. I came to ask if they were still doing this. I found this to be an exceptionally shady practice. They took my information first, then they wanted some ridiculously personal information that they didn't need. I assumed at first that I had gone to the wrong URL. I don't even know how that additional information would have helped them with their "identification". The only thing it did was expose the information of legitimate users to being stolen.

Your address and phone point to a location, not necessarily a valid identity but I understand you may be hesitant to share personal info with just anyone. If you are a EU citizen you can ask them exactly how long your data is retained and who may have access to it (it’s actually published to on their website). They use it only as a 2nd level verification to prevent spam (not shared with 3rd parties). As a German/EU company, Hetzner is subject to all regulatory requirements for handling of personal information and so when leaving, you can also request the deletion of your data (or just consult their retention period)

This is a problem with Hetzner yes. Last time they wanted a copy of my ID. I blacked out my social security number as this is considered private for Dutch citizens (even the police advises people to black this part out) and it took some arguing for them to accept it. I sent them the police advisory and that helped.

Recently I signed up for something else from Hetzner (needed temporary storage) and they didn't request anything even though I had closed my account before so I created a new one. So perhaps they have mended their ways.

Mind that sending it does not even guarantee that you'll pass that validation, I sent them my ID and they still locked me out for apparently no reason.

I'll add a +1 to managed database product. We've migrated all workloads except main application servers to Hetzner. Having a managed DB service (with backups, point-in-time recovery, etc) would have us quitting AWS in an instant.

I was considering something similar and then found this: https://gist.github.com/frozenice/fafb1565f8299a888f94d11137... (benchmarking Hetzner's cloud volumes, with unfavorable comments from people trying PG and MySQL deployments).

I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.

I would also add that on AWS, for example, you can nearly seamlessly expand EBS volumes in size without downtime. Last time I checked, not an option on Hetzner - you must take care of expanding the file system yourself. Which makes sense, as Hetzner is more basic service, but it's worth remembering that there are various differences like that, when comparing the day to day operations between such providers.

> I was considering something similar and then found this: https://gist.github.com/frozenice/fafb1565f8299a888f94d11137... (benchmarking Hetzner's cloud volumes, with unfavorable comments from people trying PG and MySQL deployments).

Network volumes always have lower iops than local disks. Thats expected, except when you pay a huge price.

> I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.

AWS GP3 has a baseline performance of 3000 iops, and from there on you have to pay for each iop/s. To reach the performance of hetzner volumes you have to add 20$/month. But with AWS you can get higher than the hetzner value. But the iops performance of the dedicated machine is not possible with gp3 (max. 16k, compared to hetzner 30-40k for local disks).

With io2 devices you can get higher than 16k iops, and higher than the hetzner local disk.

https://aws.amazon.com/de/ebs/general-purpose/

why not use Hetzner for compute and something like Planetscale with it?

Thank you for sharing your experience with Hetzner's US Cloud product. It's great to hear that you were able to find a provider that offers reduced latency and is significantly less expensive than Azure and AWS. The ability to quickly create and manage instances with SSH keys is definitely a plus, and the ability to easily scale up as needed is a valuable feature. It's also interesting to hear that you were able to migrate away from AWS RDS thanks to the use of Litestream and Cloudflare R2. Overall, it sounds like Hetzner's US Cloud product is a great value for those looking for a cost-effective and efficient cloud solution.

Doing the exact same for some side projects. It's been a delight.

Hetzner Cloud is great, but I do wish they'd offer dedicated servers in the US as well, even if it was just a fraction of their European offering.

Hetzner Cloud pricing is great, but their dedicated servers are even cheaper. For example, they offer a 16-core (32-vCPU) Ryzen 9 5950X for €103 with 128GB RAM and 2 x 3.84 TB NVMe SSDs. They offer a cloud server with 32 vCPU, 128GB RAM, and 600GB of storage for €296 - nearly triple the price and those CPU cores are probably not as good since they're likely Zen2 cores rather than the Zen3 cores of the Ryzen 9 5950X.

https://www.cpubenchmark.net/cpu.php?cpu=AMD+EPYC+7502P

https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+9+5950X

I know, benchmarks aren't everything, but the Ryzen is getting 1,432 per vCPU while the EPYC is getting 766.

The AX51 costs €59 for 8 Zen2 cores (16 vCPU) and 64GB RAM while the CCX41 costs €154 for 16 vCPU and 64GB RAM.

I know, the cloud servers come with flexibility, hourly billing, and no set-up fees. I also know that their cloud pricing is very good. Still, I wish I could get a few AX101s in the US. 3 AX101 servers would be €310/mo, each with 16 Zen3 cores, 128GB RAM, and 2x 3.8TB of storage.

Their AX line has 32 core, 64 thread AMDs for ~200 dollars/euro. That is unmatched by any major provider. So, so attractive.

https://www.hetzner.com/dedicated-rootserver/matrix-ax