News for hackers. Stuff that matters.
Re:Hetzner Object Storage by fabian2k on Monday October 07, 2024 @09:03AM (#41765600)
One thing I noticed is that some limits seem rather low, I'm not sure if this is for the beta only or by design. Particularly the 1 Gbit/s limit per bucket and the 10 buckets for all projects.

Is it typical to have such per-bucket limits? This means that you might have to take them into account when deciding on how to organize the data to avoid bottlenecks.

Re:Hetzner Object Storage by phil21 on Monday October 07, 2024 @09:07AM (#41765619)
Having implemented object storage at a service provider - it's extremely rare for any customer to have 1gbit/sec sustained throughput on an account, much less a bucket.

Those that do likely will already have an actual sales rep and the means to get these restrictions lifted, with a direct line to engineering.

Hetzner certainly markets to a different crowd, which I suspect is why these limits are being put in place ahead of time. It's very difficult to design around noisy neighbor, and will take some battle tested time in production to get right for the "average Joe" style customer who won't engage engineering resources to work with a provider to avoid hot spots and impacting other customers.

Re:Hetzner Object Storage by fabian2k on Monday October 07, 2024 @09:12AM (#41765655)
I'm more concerned about bursts, not sustained throughput. When handling large files a single client can load at 1 Gbit/s if the other side supports that.

My suspicion was also that they're intentionally targeting a different crowd than S3 and focusing on their niche here.

Re:Hetzner Object Storage by phil21 on Monday October 07, 2024 @10:13AM (#41766208)
Definitely agreed. I would be pretty surprised if this was burst capacity though. I have no insight as to how Hetzner implemented their object storage, but in general terms it's pretty difficult to ratelimit burst without adverse effects. Sustained is a lot easier to implement since you are typically having to have multiple load balancers coordinate to achieve such results. Attempting to do this at sub-minute time intervals is pretty tricky at scale.
Re:Hetzner Object Storage by seego on Monday October 07, 2024 @12:07PM (#41767398)
Just got a reply from them regarding the bucket amount and permission amount limits and both are in fact beta limits. No idea about the bandwidth though.
Re:Hetzner Object Storage by jsheard on Monday October 07, 2024 @09:20AM (#41765715)
Presumably it's built on top of their existing VPS and dedicated server boxes, which AFAIK only have 1gbit uplinks, so they'd only be able to exceed that speed when the bucket is split between multiple boxes.
Re:Hetzner Object Storage by mjochim on Monday October 07, 2024 @09:32AM (#41765832)
Hetzner VPS hosts have 10 GBit links.
Re:Hetzner Object Storage by jsheard on Monday October 07, 2024 @09:34AM (#41765855)
Huh you're right, they say their VPSes have 10gig but their dedicated boxes only have 1gig, even the really expensive ones. That seems backwards.
Re:Hetzner Object Storage by jorams on Monday October 07, 2024 @10:22AM (#41766286)
Their VPS hosts have a 10GBit uplink, but they offer no bandwidth guarantees per VPS and state you can expect 300-500MBit/s[1]. Their dedicated servers have a 1GBit uplink with guaranteed 1Gbit/s bandwidth.

[1]: https://docs.hetzner.com/cloud/technical-details/faq#what-ki...

Re:Hetzner Object Storage by ChocolateGod on Monday October 07, 2024 @10:30AM (#41766383)
The Hetzner Cloud ARM instances I believe are on 2.5Gb or 1Gb links. I've never been able to get > 1Gbit on the ARM instances, but easy getting it on x86_64.
Re:Hetzner Object Storage by pi-rat on Monday October 07, 2024 @11:49AM (#41767193)
We had a fleet of dedicated hetzner boxes w/10 gbit, it’s just an option you get and pay a little extra for. Generally had good performance as well.
Re:Hetzner Object Storage by derhuerst on Monday October 07, 2024 @09:45AM (#41765962)
It looks like you have to manually add a 10G uplink to dedicated boxes: https://docs.hetzner.com/robot/dedicated-server/network/10g-...
Re:Hetzner Object Storage by clan on Monday October 07, 2024 @09:56AM (#41766066)
Not quite correct. Their dedicated servers base configuration have 1gig. But when ordering you can add 10gig as an easy option (or add later). The 1 gig is too cheap to meter with free traffic. If you choose 10gig you pay for the traffic. Those who need it might use it :-)
Re:Hetzner Object Storage by alex23478 on Monday October 07, 2024 @02:02PM (#41768948)
At the Hetzner Summit two a few weeks ago they presented the servers used for this.

I am not sure about the exact specifications anymore, but they are building dedicated servers for this with custom chassis where each server has a ton of drives and I think they each had 40 GBits networking. These are special servers that are not available to customers directly.

Re:Hetzner Object Storage by diggan on Monday October 07, 2024 @09:14AM (#41765671)
I'm guessing that's because it's in beta, they want to set some low limits and incrementally increase it, is my bet.