News for hackers. Stuff that matters.
Re:Internet Archive: Security breach alert by marviel on Wednesday October 09, 2024 @05:53PM (#41793098)
https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...

I found this reddit thread from /r/DataHoarder about backing up the internet archive particularly interesting, given the circumstances

Re:Internet Archive: Security breach alert by numpad0 on Wednesday October 09, 2024 @06:02PM (#41793174)
50 PB * $0.014/GB = $0.7M. $0.014/GB is from[1], bare drive cost without chassis, power, or redundancy.

1: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/

Re:Internet Archive: Security breach alert by Aachen on Wednesday October 09, 2024 @06:19PM (#41793355)
How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually
Re:Internet Archive: Security breach alert by codemac on Wednesday October 09, 2024 @07:14PM (#41793831)
> How long does an average hard drive last?

This is a great question, and a state of the art kind of thing.

HDDs are sold with a lifetime drive read/write amount and power cycle warranty, along with usually some environmental operating envelope. read/write relates to the quality/space of the platter, power cycle is usually the actuator & read/write head being reseated/wearing out. Environment is the same as all other devices in a DC.

Most folks replace drives when they die (reads/writes stall or return garbage), or when the warranty runs out. Some will pay for a warranty exception, and some will just use the drive outside of warranty. Depending on how you use the drive, what environment it's in, etc changes how much you can push things.

I'd say anywhere from 4-8 years, depending on how it's used. In many cases it can be cheaper to have a worse environment for your fleet (thus using less power on hvac) and replace devices more frequently.

Re:Internet Archive: Security breach alert by bigiain on Wednesday October 09, 2024 @07:28PM (#41793922)
There's also the fact that hard drive capacities keep increasing and increasing significantly faster that the power required, and sooner or later for very long term storage it'd become cheaper to migrate all your data from those 5 year old 4TB drives to more modern 16TB ones. That's assuming you want hot access to the data and don't plan on spinning them down as soon as you've written to them, like you'd do for a cold backup of the whole IA.

I remember for a long time (I'm talking 20-ish years back here), every hard drive I bought had double or more the capacity of every drive I'd ever bought previously combined. My first ever 40MB (yes, megabyte) drive got upgraded to an 80MB one, that got updated to a 250MB one, then a 750MB, and then a whopping 2GB drive (how would I _ever_ fill that up???) - and so on. That's slowed down some, but I'm currently starting to think about upgrading my 8TB drives (Raid1 pair) with 20TB drives when the prices start to drop a bit more.

Re:Internet Archive: Security breach alert by codemac on Friday October 18, 2024 @02:51PM (#41882343)
Just don't forget that IO matters.

Drives do 140-220MB/s depending on the LBA distance of the readhead, and that's not really changing. 160MB/s is very common.

So your 8TB drives, assuming 1MiB writes with a 20ms latency and 160MB/s, you can rewrite the drive ~155 times/year. At 20T this drops to ~62 times/yr.

Re:Internet Archive: Security breach alert by ajxs on Wednesday October 09, 2024 @09:47PM (#41794709)
> ...or when the warranty runs out.

Do people really replace their drives when the warranty runs out? Hard drive manufacturers won't provide data recovery on drives that fail under warranty[1]. It makes more economical sense to just run a drive until it dies. You'll end up paying the price for a new drive either way, but less often if you ignore the warranty expiring.

1: I discovered this myself when a Seagate drive containing some important data failed under warranty. If you're foolish enough to send them a failed drive with data you need recovered (like I was), all they'll do is throw it in the bin and send you a replacement drive.

Re:Internet Archive: Security breach alert by acaiblue44 on Wednesday October 09, 2024 @10:02PM (#41794786)
I'm a new user, is this a good time to plug my project that hopes to put 200 GB on a piece of paper? https://sourceforge.net/u/acaiblue44/blog/2024/09/gigapaper0...
Re:Internet Archive: Security breach alert by mdaniel on Thursday October 10, 2024 @09:55AM (#41798864)
> I learned that I can't use file i/o in a function outside the main, which is an unspoken rule that no tutorial elucidated.

is for sure not true, that would be crazypants

Re:Internet Archive: Security breach alert by acaiblue44 on Thursday October 10, 2024 @01:16PM (#41801029)
I tried for 6 weeks. Eventually, it just stops functioning. The same program and arguments spits out "segmentation fault" 33% of the time I run it, with the other 67% working perfectly. The only way I could explain it was that it was in a function outside the main, because when I put the exact same code in the main, compiled and ran, it worked.

I have no other explanation. At some point, having too many nested loops and variables causes segmentation faults, whereas less complex code functioned without error. I needed to have certain things performed, and it only functioned in the main.

Re:Internet Archive: Security breach alert by klibertp on Thursday October 10, 2024 @05:42PM (#41803879)
Why would you try to do this in C of all languages? It's one of the worst choices, especially for a self-learner and a beginner like you. Consider: choosing another language could, on its own, 100% eliminate any possibility of getting a segfault! With just that, you'd be spared from having to produce an abomination of many thousands of loc inside a single function, which is never (unless you're Donald Knuth) a good programing practice.
Re:Internet Archive: Security breach alert by numpad0 on Thursday October 10, 2024 @07:50PM (#41804753)
Python is slower but easier, and less likely to segfault out of blue! You don't even have to have a main() loop. If you just have an idea worth demoing quick, I'd recommend switching to Python 3.
Re:Internet Archive: Security breach alert by remix2000 on Thursday October 10, 2024 @01:39PM (#41801244)
It's obviously just skill issue
Re:Internet Archive: Security breach alert by meindnoch on Thursday October 10, 2024 @04:58PM (#41803534)
What the fuck am I reading?
Re:Internet Archive: Security breach alert by bigiain on Wednesday October 09, 2024 @07:16PM (#41793843)
If this is a backup, you don't need it to be powered up and available 24x7.

So the question becomes more like "how long does an average hard drive last while powered down and still reliably be able to power back up and be read?".

I'm fairly sure that is a lot longer than the single digit years that'd be the probably answer to your question.

I wonder if there are useful guidelines for long term storage of powered down hard drives? My gut feel is the major failure modes would be electrolytic capacitor failure, bearings sticking as the lubrication ages, and obseleting of the interfaces. I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago?

Re:Internet Archive: Security breach alert by fn-mote on Wednesday October 09, 2024 @07:57PM (#41794115)
> I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago?

Easy… that original Mac is sitting in my basement and it worked like a charm last time it was powered on 4 years ago.

Re:Internet Archive: Security breach alert by jolmg on Wednesday October 09, 2024 @07:22PM (#41793892)
For archival, if you use tape, it comes out cheaper (~225k) and ought to last longer (~30 years).
Re:Internet Archive: Security breach alert by nephanth on Wednesday October 09, 2024 @07:40PM (#41793995)
For that purpose you might want to use magnetic tape like they use in long term archival services

They are cheaper per Gio, and last significantly longer

Re:Internet Archive: Security breach alert by viraptor on Wednesday October 09, 2024 @07:38PM (#41793980)
Backblaze keeps good stats. https://www.backblaze.com/cloud-storage/resources/hard-drive...

1.71% a year failure rate if you care for the hardware as much as they do.

Re:Internet Archive: Security breach alert by hsbauauvhabzb on Wednesday October 09, 2024 @08:14PM (#41794228)
But that number would increase year on year, a 10 year old drive is far more likely to fail than a 1 year old drive
Re:Internet Archive: Security breach alert by viraptor on Wednesday October 09, 2024 @08:23PM (#41794284)
Internet archive is going for long enough that I'd expect it to stabilise by now. If you replace enough of the drivers, you get a good mix.
Re:Internet Archive: Security breach alert by everforward on Wednesday October 09, 2024 @07:07PM (#41793766)
I actually find that fairly tame. For a point of comparison, Wikipedia gets ~$150M in revenue a year, an "asset rise" (I presume this is what non-profits call profit?) of ~$15M a year, and is sitting on about a quarter billion in the bank.

Not that they want to, but I think Wikipedia could fund this using their current donations if they wanted. Hell, I almost wonder if one of the big storage providers would do it for free if they could do it in their staging environment so they get real traffic. It would be less good than real backups, but extra copies are still extra copies even if they're unreliable.

Re:Internet Archive: Security breach alert by Aachen on Wednesday October 09, 2024 @07:36PM (#41793973)
You're right, I guess it is tame and achievable so far as organisations go. I was imagining trying to get some friends together to have a decent percentage of the IA backed up, but that seems out of reach based on this napkin math. Not that that is necessarily demotivating, but it's going to depend on a lot of people intuitively seeing the value and keeping up their share
Re:Internet Archive: Security breach alert by everforward on Thursday October 10, 2024 @06:43PM (#41804333)
Yeah, as a sort of pet project I don’t think backing up the whole thing is possible.

You might be able to back up a significant portion of the unique data in IA if you limited it to text files. I think they probably have the highest information to file size ratio.

It’s also probably the most likely to already be back up, though. Interesting issue; you might also get somewhere by cutting the 50TB up into 10GB torrents (or 100GB or whatever, something reasonable for a consumer hard drive) and maybe adding a script that checks the torrent swarm stats to recommend a torrent to download.

Something where I run it, tell it I want to let it use 600GB, and it hands me torrent files for the least seeded 600GB. Maybe a super basic web UI so people can see how well backed up it is?

Unsure if people would sign on or not; I probably would. I’ve got 10 or so TB of NFS I’m not using I could chuck at it. I would guess there are other data hoarders out there who would do the same, but only if it were somewhat easy. I’m probably not going to volunteer to do an hour of rtorrent cleanup a week to make sure I’m backing up the right things.

Re:Internet Archive: Security breach alert by rolandog on Friday October 11, 2024 @04:59AM (#41807668)
I think a part of the scope of this project may have already been solved by The FreeNet Project (now HyphaNet) [0].

[0]: https://www.hyphanet.org/

Re:Internet Archive: Security breach alert by Intralexical on Wednesday October 09, 2024 @08:11PM (#41794208)
They should probably consider it, really.

A good portion of the text on Wikipedia relies on Wayback Machine links to remain verifiable. If they lose that, I guess the editors might have to comb every page for information which would need to be either resourced or deleted.

Re:Internet Archive: Security breach alert by tivert on Wednesday October 09, 2024 @07:17PM (#41793851)
> How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually

You'd have to spend a lot more, because with that many drives, you need redundancy now.

Re:Internet Archive: Security breach alert by Aachen on Wednesday October 09, 2024 @07:38PM (#41793983)
True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once
Re:Internet Archive: Security breach alert by tivert on Wednesday October 09, 2024 @07:45PM (#41794030)
> True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once

I think with that many drives, you'd be losing them constantly, and I suppose you wouldn't know which ones until later (assuming you're doing an offline backup, if you aren't you have to factor in power costs).

Re:Internet Archive: Security breach alert by ks2048 on Wednesday October 09, 2024 @08:43PM (#41794401)
Interesting to compare their stated drive $/GB to their B2 offering: $6/TB/mo for "pay-as-you-go",

hard-drive price: $0.014/GB

B2 price (12*6/1024): $0.070/GB/year

Re:Internet Archive: Security breach alert by alwayslikethis on Wednesday October 09, 2024 @08:49PM (#41794444)
Electricity, bandwidth, and generally running a business is not free. Also for these pay-as-you-go setups you'd need a considerable amount of free space available on demand. That said, it's not an especially cheap option. Hetzner has storage boxes for EUR 2.5/TB/mo (in fixed 5 and 10TB boxes though)
Re:Internet Archive: Security breach alert by ks2048 on Wednesday October 09, 2024 @09:30PM (#41794642)
Yeah, I wasn't trying to point out that it's a bad price. I think it's pretty good: same price for two years with all the maintenance.
Re:Internet Archive: Security breach alert by PostOnce on Wednesday October 09, 2024 @10:06PM (#41794812)
IA stores lots of redundant stuff in 5 file formats and none of them are particularly well-compressed, I think. There are (big) savings to be had, but maybe figuring that out (software dev and compute time) isn't worth it?
Re:Internet Archive: Security breach alert by nikisweeting on Wednesday October 09, 2024 @06:35PM (#41793499)
It's been tried several times, but it's hard because it's such a massive quantity of data. The IPFS backup never really got off the ground.

They have their own backups which I think is good enough for now unless someone plans on donating a few hundred million.

Re:Internet Archive: Security breach alert by vincentpants on Wednesday October 09, 2024 @06:51PM (#41793653)
Oh no! I didn't know their IPFS initiative didn't pan out. What happened to it? I am surprised how hard it is to google. I remember interviewing for a role on that team at the archive to help move it to filecoin. Was so happy to hear that the effort was underway to decentralize their datastore. We need this more than ever.
Re:Internet Archive: Security breach alert by nikisweeting on Wednesday October 09, 2024 @07:15PM (#41793837)
There are people still working on trying to make it happen but it's just a collosal amount of data and filesystems are notoriously hard, so it's very slow going.

From my own personal experience doing distributed archiving with no relation to Archive.org, Filecoin/IPFS's UX isn't quite there yet. They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs.

That's why I still haven't integrated ArchiveBox with IPFS/Filecoin/Storj, let my data live in a normal filesystem dammit!

Re:Internet Archive: Security breach alert by Aachen on Wednesday October 09, 2024 @07:24PM (#41793900)
> They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs.

I don't understand this part. What data would you have to give them? Why can't it just live next to your stuff on your OS' filesystem?

Re:Internet Archive: Security breach alert by dannyobrien on Wednesday October 09, 2024 @08:17PM (#41794248)
For IPFS, I'm fairly sure you can now serve from your normal filesystem, rather than load it into their blockstorage -- or at least the blockstorage has pointers to real data blocks that are part of your existing files (it's the nocopy option[1]; it's marked as experimental, so there may be some sharp edges.)

For Filecoin, if you want fast access, you do need to keep a second hot plaintext copy, as well as the sealed Filecoin copy. But that works for the backup case for IA, because the hot copy would be served from the archive's existing infrastructure (and/or a distributed IPFS hot cache) -- you'd just use Filecoin for the proven safe backup.

The project to back up IA to Filecoin is still ongoing. The IA dashboard that shows the current state is (perhaps predictably) down at the moment, but it crossed the 1PiB line last year[2], and they've been optimising the onboarding flow recently.

[1] https://docs.ipfs.tech/reference/kubo/cli/#ipfs-add

[2] https://blog.archive.org/2023/10/20/celebrating-1-petabyte-o...

(Disclosure: I work at the Filecoin Foundation/Filecoin Foundation for the Decentralized Web, which partners with the Archive on this project, as well as supporting other Internet Archive backup projects.)

Re:Internet Archive: Security breach alert by nikisweeting on Wednesday October 09, 2024 @08:50PM (#41794449)
Needing to keep a separate hot copy at 220PiB is already ~$7M/yr, and multiples much more than that if you factor in labor and redundancy. The --nocopy option looks great though, I didn't see it last time I was looking around for an MFS/FUSE solution, I'll try it.

I appreciate your effort and I hope the project continues.

Re:Internet Archive: Security breach alert by nightpool on Wednesday October 09, 2024 @07:35PM (#41793964)
They're saying that the client software (the servers that speak the IPFS protocols) has to load the files to be served into their own local storage database, it can't just keep a "metadata file" and read the existing files off disk. Presumably somebody could write a client that spoke the IPFS protocol and did this, or fork the main Go or JS one, but until someone does that they're stuck with the software that's already been written
Re:Internet Archive: Security breach alert by pshc on Wednesday October 09, 2024 @08:16PM (#41794245)
IPFS is all content-hash-addressed, so my guess is the IPFS service spirits the files away to a (hopefully) immutable store for the sake of sanity.
Re:Internet Archive: Security breach alert by pbhjpbhj on Wednesday October 09, 2024 @06:56PM (#41793690)
Perhaps you can persuade Elon that it owns the libs?
Re:Internet Archive: Security breach alert by nikisweeting on Wednesday October 09, 2024 @07:12PM (#41793807)
I don't want Elon anywhere near Archive.org, please don't give him any ideas. There are plenty of other people in the world with money.
Re:Internet Archive: Security breach alert by bunabhucan on Wednesday October 09, 2024 @08:52PM (#41794458)
"Based on historical records from the first half of the last century, Mr Musk (inventor of the car and the rocket) and President Xi were the most respected and popular individuals on earth."
Re:Internet Archive: Security breach alert by gaudystead on Wednesday October 09, 2024 @11:45PM (#41795302)
History is written by the winners...
Re:Internet Archive: Security breach alert by aryonoco on Thursday October 10, 2024 @05:51AM (#41797231)
Maybe in the immediate aftermath, but not long after. King Leopold "won" but we now all think he was terrible.
Re:Internet Archive: Security breach alert by yard2010 on Wednesday October 09, 2024 @09:50PM (#41794724)
Yes please, we need this lunatic out of our life, not the other way around
Re:Internet Archive: Security breach alert by creer on Thursday October 10, 2024 @12:45AM (#41795594)
Backup / duplication is not an easy project for sure. But meanwhile for now IA is a single organization operating under one legal system. And one technical setup, would be relevant today. That's a major weakness.
Re:Internet Archive: Security breach alert by EamonnMR on Thursday October 10, 2024 @03:30PM (#41802575)
Suppose we each backed up sites we cared about rather than trying to mirror the whole thing...