News for hackers. Stuff that matters.
Re:Hetzner Object Storage by bhouston on Monday October 07, 2024 @09:07AM (#41765620)
I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd only use them for stuff I can afford to lose. From the outside as a consumer of these services you do not know how robust they actually are on the inside.

I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:

https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-dam...

Re:Hetzner Object Storage by bilekas on Monday October 07, 2024 @10:08AM (#41766177)
Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery.

From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.

Re:Hetzner Object Storage by voytec on Monday October 07, 2024 @10:48AM (#41766548)
Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.
Re:Hetzner Object Storage by liotier on Monday October 07, 2024 @11:27AM (#41766987)
Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player.

A great writeup of the incident and its context, three years afterwards: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...

Re:Hetzner Object Storage by ChocolateGod on Monday October 07, 2024 @06:10PM (#41771614)
> But nowadays they are just as mainstream industrial as their competitors

Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google.

It's slow slow slow slow slow and horribly buggy, did I mention it's slow?

Re:Hetzner Object Storage by lopkeny12ko on Monday October 07, 2024 @10:49AM (#41766563)
People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...
Re:Hetzner Object Storage by JoshTriplett on Monday October 07, 2024 @11:03AM (#41766716)
> If the building burns down in a fire, of course your data will be gone

From https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu... :

> S3 Standard, [...] redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.

Re:Hetzner Object Storage by mardifoufs on Monday October 07, 2024 @10:56AM (#41766650)
Most of the "big" cloud providers aren't as negligent as OVH though. That entire datacenter was just one blunder after the other (wood structures, barely functional fire suppression, no proper power shut down, insane replication strategy that caused most of the replicas to burn in the very same fire in the same building...)
Re:Hetzner Object Storage by numbsafari on Monday October 07, 2024 @09:28AM (#41765799)
"Cloud 3-2-1" to the rescue...

I take the typical formulation (e.g., [1]), and translate it into:

- Keep 3 copies of your data: production + 2.

- Keep 2 snapshots, stored separately.

- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.

It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.

[1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

Re:Hetzner Object Storage by chasd00 on Monday October 07, 2024 @09:46AM (#41765977)
I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that.

another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.

Re:Hetzner Object Storage by latchkey on Monday October 07, 2024 @11:53AM (#41767247)
Switch builds their DCs with two roofs...

https://www.switch.com/switch-shield/

The Switch one I'm in (Grand Rapids Pyramid), the servers are about 3/4's underground.

Re:Hetzner Object Storage by timenova on Monday October 07, 2024 @09:45AM (#41765965)
I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested.

However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.

[0] https://www.tigrisdata.com

[1] https://www.tigrisdata.com/docs/migration/#starting-the-migr...

Re:Hetzner Object Storage by rmbyrro on Monday October 07, 2024 @11:09AM (#41766774)
Backblaze is 3x less expensive and offers more generous free allowances.
Re:Hetzner Object Storage by ChocolateGod on Monday October 07, 2024 @06:08PM (#41771575)
> Access small objects at close to Redis speed

This is a massive claim that unless they're storing everything in memory I don't know how you can get close to Redis speed.

Even the flash-based Redis alternatives can't get close (but are good enough)

Re:Hetzner Object Storage by vidarh on Monday October 07, 2024 @09:39AM (#41765901)
S3 durability is pretty much the only thing I consider worth using EC2 for.

S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...

Re:Hetzner Object Storage by ptman on Monday October 07, 2024 @10:04AM (#41766140)
IIRC Cloudflare R2 is designed as a cache for S3 and has free egress

edit: apparently not writethrough: https://developers.cloudflare.com/r2/data-migration/sippy/

Re:Hetzner Object Storage by jgrahamc on Monday October 07, 2024 @10:16AM (#41766246)
No, R2 is not an "S3 cache". It's a replacement for S3 with no egress fees.
Re:Hetzner Object Storage by viraptor on Monday October 07, 2024 @04:04PM (#41770361)
Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready.

Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.

Re:Hetzner Object Storage by guerby on Monday October 07, 2024 @10:22AM (#41766294)
It just caught those who rely on one provider, which is never a good practice.

rclone to somewhere else at least.

And 3-2-1 rule for backups if you're serious.

Re:Hetzner Object Storage by fabian2k on Monday October 07, 2024 @10:41AM (#41766473)
I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.
Re:Hetzner Object Storage by smartbit on Monday October 07, 2024 @09:40AM (#41765913)
Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.
Re:Hetzner Object Storage by sangnoir on Monday October 07, 2024 @03:05PM (#41769714)
As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.
Re:Hetzner Object Storage by 42lux on Monday October 07, 2024 @09:22AM (#41765737)
You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.
Re:Hetzner Object Storage by clarkdave on Monday October 07, 2024 @09:27AM (#41765789)
> It's not like there is a mirror datacenter just two blocks away

Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options)

It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region

[0] https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform

Re:Hetzner Object Storage by ahofmann on Monday October 07, 2024 @09:59AM (#41766095)
It literally was one building in france, that burned to the ground: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
Re:Hetzner Object Storage by abadpoli on Monday October 07, 2024 @10:41AM (#41766474)
If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it.

OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.

Re:Hetzner Object Storage by t0mas88 on Monday October 07, 2024 @03:42PM (#41770113)
Is the any GCP documentation on that? Sounds like a far bigger risk going with GCP than AWS if all zones in a region are in the same building.
Re:Hetzner Object Storage by luuurker on Monday October 07, 2024 @02:08PM (#41769032)
One of my VPS was lost in that fire, but since my backups where somewhere else, I didn't lose much.

The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.