News for hackers. Stuff that matters.
Re:ZFS 128 bit storage: Are you high? (2004) by mrb on Monday February 12, 2018 @09:06PM (#16364197)
They had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 years. In 2005 I bought 250GB HDDs for $110 each. In 2018 you can find 8TB HDDs for $150 each. That's 5 doublings.

So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-...

So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.

Re:ZFS 128 bit storage: Are you high? (2004) by bastawhiz on Monday February 12, 2018 @10:30PM (#16364511)
It's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system.

If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.

Re:ZFS 128 bit storage: Are you high? (2004) by Moter8 on Tuesday February 13, 2018 @04:12AM (#16365759)
>If 64 bits isn't enough, the next logical step is 128 bits. >That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem

;)

Re:ZFS 128 bit storage: Are you high? (2004) by AlphaSite on Tuesday February 13, 2018 @12:03PM (#16368691)
HFS stuck around for around 30 years as well. And is still shipped on some systems.
Re:ZFS 128 bit storage: Are you high? (2004) by ianai on Monday February 12, 2018 @11:20PM (#16364724)
And let’s not forget all the crazy (better word?) features like independence from indianness. They built the file system with an eye toward versatility.
Re:ZFS 128 bit storage: Are you high? (2004) by mnw21cam on Tuesday February 13, 2018 @04:12AM (#16365756)
> indianness

That has to be one of the worst typos I have ever seen.

Re:ZFS 128 bit storage: Are you high? (2004) by arghwhat on Tuesday February 13, 2018 @04:27AM (#16365837)
The filesystem does not share any culture with Native America and India.
Re:ZFS 128 bit storage: Are you high? (2004) by MatthewWilkes on Tuesday February 13, 2018 @09:24AM (#16367291)
The term does derive from Gullivers Travels, where the big endians and the little endians were both to be found on an island in the Indian ocean.

Of course, the real reason for the mistake is that phone spelling correction will convert endian to Indian.

Re:ZFS 128 bit storage: Are you high? (2004) by ianai on Thursday February 15, 2018 @04:44PM (#16387975)
It felt wrong at the time but it’s been too long for me to remember the correct spelling.
Re:ZFS 128 bit storage: Are you high? (2004) by gonzo on Monday February 12, 2018 @11:09PM (#16364670)
In 2016, Backblaze swapped out more than 3,500 2 TB HGST and WDC hard drives for 2,400 8 TB Seagate drives.

That’s 7 PB to 19.2 PB. If they had replaced all 3,500 drives it would be 4X.

Re:ZFS 128 bit storage: Are you high? (2004) by mbreese on Tuesday February 13, 2018 @04:23AM (#16365807)
But that's not all running under a single filesystem. It's multiple file systems that are orchestrated together to form their network storage space.

I think that's the point from some of the other comments (below). Instead of having every larger and larger single filesystems, once you get to a certain point (~1-2PB is my guess), it simply isn't worth it to organize your data under a single filesystem.

That's not to say ZFS was wrong in choosing 128 bits... at the time ZFS was designed, the level of horizontal partitioning that we are now accustomed to just wasn't a thing. Compared to HDD times, networks were far too slow to support such a thing. Now, however, we talk about HDDs being too slow compared to our networks. At the time, 128 bits was a solid choice, even if it was overkill.

Plus, Sun was definitely a scale up kind of company... if you want to get bigger, by a bigger ($$$$) box. So having a large capacity FS was good for their bottom line. Now, we think in terms of scaling out with multiple (cheaper) smaller boxes... which is probably the only way we've managed with current levels of data.

Re:ZFS 128 bit storage: Are you high? (2004) by arghwhat on Tuesday February 13, 2018 @04:51AM (#16365958)
> It's multiple file systems that are orchestrated together to form their network storage space.

No, that is not true. A vault acts as a single unit, effectively making it a single filesystem as much as a ZFS pool is a single filesystem.

Internally, a vault operates on a set of "tomes", which are groups of 1 harddrive from each storage pod (20). These "tomes" handle the redundancy aspect as well. However, the "tome" is not the filesystem.

The concepts map cleanly to the ZFS world: A vault is equivalent to a ZFS pool, and a tome is equivalent to a ZFS vdev in redundant configuration. Neither vdevs nor tomes are filesystems on their own—they only become a filesystem when combined into pools and vaults, respectively.

Now, your argument does hold if interpreted differently: One does not use a normal filesystem for those absurd sizes. However, it is a single filesystem.

Re:ZFS 128 bit storage: Are you high? (2004) by mbreese on Tuesday February 13, 2018 @09:41AM (#16367427)
Yes, I had forgotten that they striped their data across multiple pods... I'm not familiar with how each Pod of their works. Is it really reading/writing directly to the devices, or is there an intermediate FS that is storing the files (well, chunks of files) and the Backblaze software is orchestrating it? Or do the 20 pods have one central interface?

I guess I was assuming it worked like Lustre or BeeGFS where there were smaller FSs (ext4, XFS) at play that handled storing file (chunks) and then a higher level interface that managed it all.

Either way -- the original comment was referring to Backblaze swapping 7PBs for 19PBs. Which we agree is not a normal size for a "normal" file system! ZFS is a strange beast... it was designed to support a world that just never showed up. It is a "normal" FS that works from laptops to servers, but was designed to support a big iron world that never really appeared (much). Instead, here we are comparing it not to XFS, but to cloud FS's that were designed for a completely different "cloud" world.

Re:ZFS 128 bit storage: Are you high? (2004) by arghwhat on Tuesday February 13, 2018 @11:55AM (#16368615)
The vault is the lowest unit you can interact with. There is some routing on top which picks a vault for you to use, but that's about it.

I would assume that the tome, beneath network protocols, interacts directly with the individual block devices. They have implemented things like Reed-Solomon error correction themselves to handle redundancy across the tome (which to recap is their cross-server vdev equivalent), so they have indeed implemented tasks of a filesystem and RAID manager on top of their lower-level primitives.

I also believe that what their low-level primitives are is actually a rather irrelevant implementation detail. A filesystem presents a way to store and organize files, and is a filesystem regardless of whether it is implemented on a block device or an egg-engraver. It could be using FAT16 as object store, sharding across files and handling metadata elsewhere. I would still consider such setup to be a true filesystem, as it is not simply a 1:1 network-to-disk protocol.

And yes, ZFS won't see the large systems it is designed to handle anytime soon. What Sun did not see coming was the death of large, high-capacity servers, and the rise of distributed solutions. But then again, no one saw that coming in 2001 when ZFS development started.

However, considering that storage sizes will continue to increase, who knows what will happen in the future... 128 bit filesystems might come in handy at some point.

(Note that my information on backblaze's systems is based on public information—I am not an insider of any sort.)

Re:ZFS 128 bit storage: Are you high? (2004) by Sean1708 on Tuesday February 13, 2018 @04:58AM (#16365986)
> it simply isn't worth it to organize your data under a single filesystem

Is that a fundamental thing or is it just because there aren't many filesystems that can handle it? It seems to me like a single filesystem would be conceptually easier, but I'm far from knowledgeable in this area.

Re:ZFS 128 bit storage: Are you high? (2004) by mbreese on Tuesday February 13, 2018 @09:54AM (#16367516)
Comparing the amount of effort that has been spent on distributed filesystems (where everything is managed under one hierarchy) to the usage of such systems, I'm going to say that it's a fundamental law of trying to organize things.

But really, I do think it's physics at play... or rather logistics.

Let's say you're trying to put together a 10PB storage system in a datacenter. If you're using 60 disk JBODs with 10TB drives, thats 600TB of raw space per 4U (or a max of 6PB per rack, or 12PB in 2 racks). Now realistically due to power, network gear, etc... instead of maxing out 2 racks, you'd probably split that into at least 4 (or more) racks. If you're Backblaze (see above), you split those 20 pods across 20 separate racks (and add more volumes, but that's another story).

But once you're at that level, for redundancy, you'd not only managing which disk data is written/replicated to, but also the JBOD/pod, and the rack (and the data center). So, at that point, you're not going to working with a "normal"/traditional file system. You could with ZFS define redundancies that would work like that, but realistically, because of the extra overhead, you're better off changing how you organize your data.

(Plus: ZFS isn't very good at expanding a filesystem with new storage. You can do it, but your pool ends up unbalanced and performance suffers. When you decide to have a 1+PB system, you normally want to have the ability to add more storage space, which means a different type of FS)

Re:ZFS 128 bit storage: Are you high? (2004) by jerven on Tuesday February 13, 2018 @05:17PM (#16371355)
Just hit df -lh on the EBI login nodes and see a 22PB filesystem at 93% full. I suspect if I dig deeper I will find a bigger one as they had 120PB of storage last year[1].

[1]https://academic.oup.com/nar/article/46/D1/D21/4658830