News for hackers. Stuff that matters.
Re:ZFS 128 bit storage: Are you high? (2004) by vbezhenar on Monday February 12, 2018 @05:58PM (#16363163)
> Some customers already have datasets on the order of a petabyte, or 2^50 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away. Moore's Law for storage predicts that capacity will continue to double every 9-12 months, which means we'll start to hit the 64-bit limit in about a decade. Storage systems tend to live for several decades, so it would be foolish to create a new one without anticipating the needs that will surely arise within its projected lifetime.

So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year.

Re:ZFS 128 bit storage: Are you high? (2004) by mrb on Monday February 12, 2018 @09:06PM (#16364197)
They had to be on the conservative side, so they assumed a doubling every 9-12 months. In reality we have seen one doubling every ~34 months in the last 14 years. In 2005 I bought 250GB HDDs for $110 each. In 2018 you can find 8TB HDDs for $150 each. That's 5 doublings.

So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-...

So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.

Re:ZFS 128 bit storage: Are you high? (2004) by bastawhiz on Monday February 12, 2018 @10:30PM (#16364511)
It's also worth noting that file system developers should think about their file system fifty to a hundred years out. Flash drives today often still come preformatted with FAT32, twenty two years after its release in 1996. Up until a couple years ago, my dad had an external drive formatted with FAT16--a thirty year old file system.

If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.

Re:ZFS 128 bit storage: Are you high? (2004) by Moter8 on Tuesday February 13, 2018 @04:12AM (#16365759)
>If 64 bits isn't enough, the next logical step is 128 bits. >That's enough to survive Moore's Law until I'm dead, and after that, it's not my problem

;)

Re:ZFS 128 bit storage: Are you high? (2004) by AlphaSite on Tuesday February 13, 2018 @12:03PM (#16368691)
HFS stuck around for around 30 years as well. And is still shipped on some systems.
Re:ZFS 128 bit storage: Are you high? (2004) by ianai on Monday February 12, 2018 @11:20PM (#16364724)
And let’s not forget all the crazy (better word?) features like independence from indianness. They built the file system with an eye toward versatility.
Re:ZFS 128 bit storage: Are you high? (2004) by mnw21cam on Tuesday February 13, 2018 @04:12AM (#16365756)
> indianness

That has to be one of the worst typos I have ever seen.

Re:ZFS 128 bit storage: Are you high? (2004) by arghwhat on Tuesday February 13, 2018 @04:27AM (#16365837)
The filesystem does not share any culture with Native America and India.
Re:ZFS 128 bit storage: Are you high? (2004) by MatthewWilkes on Tuesday February 13, 2018 @09:24AM (#16367291)
The term does derive from Gullivers Travels, where the big endians and the little endians were both to be found on an island in the Indian ocean.

Of course, the real reason for the mistake is that phone spelling correction will convert endian to Indian.

Re:ZFS 128 bit storage: Are you high? (2004) by ianai on Thursday February 15, 2018 @04:44PM (#16387975)
It felt wrong at the time but it’s been too long for me to remember the correct spelling.
Re:ZFS 128 bit storage: Are you high? (2004) by gonzo on Monday February 12, 2018 @11:09PM (#16364670)
In 2016, Backblaze swapped out more than 3,500 2 TB HGST and WDC hard drives for 2,400 8 TB Seagate drives.

That’s 7 PB to 19.2 PB. If they had replaced all 3,500 drives it would be 4X.

Re:ZFS 128 bit storage: Are you high? (2004) by mbreese on Tuesday February 13, 2018 @04:23AM (#16365807)
But that's not all running under a single filesystem. It's multiple file systems that are orchestrated together to form their network storage space.

I think that's the point from some of the other comments (below). Instead of having every larger and larger single filesystems, once you get to a certain point (~1-2PB is my guess), it simply isn't worth it to organize your data under a single filesystem.

That's not to say ZFS was wrong in choosing 128 bits... at the time ZFS was designed, the level of horizontal partitioning that we are now accustomed to just wasn't a thing. Compared to HDD times, networks were far too slow to support such a thing. Now, however, we talk about HDDs being too slow compared to our networks. At the time, 128 bits was a solid choice, even if it was overkill.

Plus, Sun was definitely a scale up kind of company... if you want to get bigger, by a bigger ($$$$) box. So having a large capacity FS was good for their bottom line. Now, we think in terms of scaling out with multiple (cheaper) smaller boxes... which is probably the only way we've managed with current levels of data.

Re:ZFS 128 bit storage: Are you high? (2004) by arghwhat on Tuesday February 13, 2018 @04:51AM (#16365958)
> It's multiple file systems that are orchestrated together to form their network storage space.

No, that is not true. A vault acts as a single unit, effectively making it a single filesystem as much as a ZFS pool is a single filesystem.

Internally, a vault operates on a set of "tomes", which are groups of 1 harddrive from each storage pod (20). These "tomes" handle the redundancy aspect as well. However, the "tome" is not the filesystem.

The concepts map cleanly to the ZFS world: A vault is equivalent to a ZFS pool, and a tome is equivalent to a ZFS vdev in redundant configuration. Neither vdevs nor tomes are filesystems on their own—they only become a filesystem when combined into pools and vaults, respectively.

Now, your argument does hold if interpreted differently: One does not use a normal filesystem for those absurd sizes. However, it is a single filesystem.

Re:ZFS 128 bit storage: Are you high? (2004) by mbreese on Tuesday February 13, 2018 @09:41AM (#16367427)
Yes, I had forgotten that they striped their data across multiple pods... I'm not familiar with how each Pod of their works. Is it really reading/writing directly to the devices, or is there an intermediate FS that is storing the files (well, chunks of files) and the Backblaze software is orchestrating it? Or do the 20 pods have one central interface?

I guess I was assuming it worked like Lustre or BeeGFS where there were smaller FSs (ext4, XFS) at play that handled storing file (chunks) and then a higher level interface that managed it all.

Either way -- the original comment was referring to Backblaze swapping 7PBs for 19PBs. Which we agree is not a normal size for a "normal" file system! ZFS is a strange beast... it was designed to support a world that just never showed up. It is a "normal" FS that works from laptops to servers, but was designed to support a big iron world that never really appeared (much). Instead, here we are comparing it not to XFS, but to cloud FS's that were designed for a completely different "cloud" world.

Re:ZFS 128 bit storage: Are you high? (2004) by arghwhat on Tuesday February 13, 2018 @11:55AM (#16368615)
The vault is the lowest unit you can interact with. There is some routing on top which picks a vault for you to use, but that's about it.

I would assume that the tome, beneath network protocols, interacts directly with the individual block devices. They have implemented things like Reed-Solomon error correction themselves to handle redundancy across the tome (which to recap is their cross-server vdev equivalent), so they have indeed implemented tasks of a filesystem and RAID manager on top of their lower-level primitives.

I also believe that what their low-level primitives are is actually a rather irrelevant implementation detail. A filesystem presents a way to store and organize files, and is a filesystem regardless of whether it is implemented on a block device or an egg-engraver. It could be using FAT16 as object store, sharding across files and handling metadata elsewhere. I would still consider such setup to be a true filesystem, as it is not simply a 1:1 network-to-disk protocol.

And yes, ZFS won't see the large systems it is designed to handle anytime soon. What Sun did not see coming was the death of large, high-capacity servers, and the rise of distributed solutions. But then again, no one saw that coming in 2001 when ZFS development started.

However, considering that storage sizes will continue to increase, who knows what will happen in the future... 128 bit filesystems might come in handy at some point.

(Note that my information on backblaze's systems is based on public information—I am not an insider of any sort.)

Re:ZFS 128 bit storage: Are you high? (2004) by Sean1708 on Tuesday February 13, 2018 @04:58AM (#16365986)
> it simply isn't worth it to organize your data under a single filesystem

Is that a fundamental thing or is it just because there aren't many filesystems that can handle it? It seems to me like a single filesystem would be conceptually easier, but I'm far from knowledgeable in this area.

Re:ZFS 128 bit storage: Are you high? (2004) by mbreese on Tuesday February 13, 2018 @09:54AM (#16367516)
Comparing the amount of effort that has been spent on distributed filesystems (where everything is managed under one hierarchy) to the usage of such systems, I'm going to say that it's a fundamental law of trying to organize things.

But really, I do think it's physics at play... or rather logistics.

Let's say you're trying to put together a 10PB storage system in a datacenter. If you're using 60 disk JBODs with 10TB drives, thats 600TB of raw space per 4U (or a max of 6PB per rack, or 12PB in 2 racks). Now realistically due to power, network gear, etc... instead of maxing out 2 racks, you'd probably split that into at least 4 (or more) racks. If you're Backblaze (see above), you split those 20 pods across 20 separate racks (and add more volumes, but that's another story).

But once you're at that level, for redundancy, you'd not only managing which disk data is written/replicated to, but also the JBOD/pod, and the rack (and the data center). So, at that point, you're not going to working with a "normal"/traditional file system. You could with ZFS define redundancies that would work like that, but realistically, because of the extra overhead, you're better off changing how you organize your data.

(Plus: ZFS isn't very good at expanding a filesystem with new storage. You can do it, but your pool ends up unbalanced and performance suffers. When you decide to have a 1+PB system, you normally want to have the ability to add more storage space, which means a different type of FS)

Re:ZFS 128 bit storage: Are you high? (2004) by jerven on Tuesday February 13, 2018 @05:17PM (#16371355)
Just hit df -lh on the EBI login nodes and see a 22PB filesystem at 93% full. I suspect if I dig deeper I will find a bigger one as they had 120PB of storage last year[1].

[1]https://academic.oup.com/nar/article/46/D1/D21/4658830

Re:ZFS 128 bit storage: Are you high? (2004) by bhouston on Monday February 12, 2018 @06:09PM (#16363227)
I think the largest file systems in the world are Amazon's S3, and maybe Glacier. I guess Google's internal system used for YouTube and Gmail probably outrank it but there are no numbers for it. But I doubt that they are in a single address space on a single file system.

So basically we moved away from singular large unified file systems and built swarms of little file systems.

Re:ZFS 128 bit storage: Are you high? (2004) by vesinisa on Monday February 12, 2018 @07:21PM (#16363680)
Indeed. "Cloud" is essentially tons of commodity hardware nodes running a tailored userspace distributed file system application. There now certainly exist cloud services that host well in excess of 2^64 bytes, but they are not hosted on traditional file systems, but an array of those.

I can't certainly blame their choice though, and 16-million terabyte hard drive arrays (the limit of 64-bit addressing) on a large mainframe are really just a few quantum leaps away. Heck, I still vividly recall buying a MASSIVE 40 GB HDD in 2000 - it seemed excessive beyond wildest dreams at the time (and was promptly filled to brim at a LAN party.)

Since, advances in mass storage tech, most importantly the successful commercialization of perpendicular recording in 2005, mean that nowadays 12 TB drives are commercially viable - roughly 1000x the size of a 40 GB drive.

In the next 20 years, petabyte-sized drives might well be available (though almost certainly not based on any form of magnetic recording). Once you hook hundred 50-petabyte drives to a single fileserver, you start closing in on limitations of 64-bit addressing.

Re:ZFS 128 bit storage: Are you high? (2004) by dsr_ on Monday February 12, 2018 @08:46PM (#16364100)
Overly pedantic quibble: a quantum of anything is the minimum possible unit. One planck length, time, mass, charge, temperature.

Everybody uses it to mean "big". Oh, language.

Re:ZFS 128 bit storage: Are you high? (2004) by taneq on Monday February 12, 2018 @09:44PM (#16364348)
"Quantum leap" is typically interpreted as "big leap" but I believe the true implication is more like "indivisible leap", in line with your quibble. A Turing-complete computer is a "quantum leap" ahead of a non-Turing-complete calculator. A warp drive is a "quantum leap" ahead of a reaction drive. You either have all of it or none of it.
Re:ZFS 128 bit storage: Are you high? (2004) by rusk on Tuesday February 13, 2018 @04:47AM (#16365932)
Quantum leap is an increase in the units, or “quanta” you use to measure something i.e. MHz to GHz - the term doesn’t relate directly to quantum mechanics
Re:ZFS 128 bit storage: Are you high? (2004) by dsr_ on Tuesday February 13, 2018 @06:01AM (#16366211)
In your example you are using the same units - Hertz, one cycle per second - and adding SI prefixes for million X and billion X.
Re:ZFS 128 bit storage: Are you high? (2004) by rusk on Tuesday February 13, 2018 @07:00AM (#16366462)
Yes, but the "quantum" has increased by order of magnitude.

The point is, that the increase in frequency is so large, that using MHz is no longer sensible.

There has been a "quantum leap" in frequency.

Re:ZFS 128 bit storage: Are you high? (2004) by quickthrower2 on Monday February 12, 2018 @11:14PM (#16364696)
I know who we should ask: Ziggy
Re:ZFS 128 bit storage: Are you high? (2004) by beobab on Tuesday February 13, 2018 @03:41AM (#16365645)
I loved that show so much! It was ages after Enterprise came out that I stopped expecting Captain Archer to say "Oh boy!" just before the main title.
Re:ZFS 128 bit storage: Are you high? (2004) by dmix on Monday February 12, 2018 @09:37PM (#16364320)
But is a "Quantum leap" really equivalent to leaping one planck? Honest question.

I assumed it implied something far 'larger' such as a leap through spacetime...

Re:ZFS 128 bit storage: Are you high? (2004) by white-flame on Monday February 12, 2018 @11:41PM (#16364816)
A quantum leap is a discontinuity, a change that cannot happen gradually.

It typically implies a revolutionary change over evolutionary improvement.

Re:ZFS 128 bit storage: Are you high? (2004) by aidenn0 on Monday February 12, 2018 @09:50PM (#16364365)
I believe the term originally applied to electron energy level transitions within an atom, which only vaguely correlates with anything one might call distance.

The interesting thing there was that the electrons existed only at discrete energy levels, and the transition between them appeared instantaneous.

Re:ZFS 128 bit storage: Are you high? (2004) by ianai on Monday February 12, 2018 @11:15PM (#16364702)
I always assumed that sprang from the tv show with that name.
Re:ZFS 128 bit storage: Are you high? (2004) by vesinisa on Tuesday February 13, 2018 @03:49AM (#16365674)
In this case, I actually used it purposefully. Hitting on quantum mechanical limits are in part holding us from infinitely increasing the information density on HDDs and SSDs.
Re:ZFS 128 bit storage: Are you high? (2004) by versteegen on Tuesday February 13, 2018 @09:43PM (#16372990)
The usage in the article is that 64/128 bits is the addressing used for the blocks. A block is 512 bytes. So a 64-bit filesystem would actually be limited to 2^73 bytes = 8192 EiB = 8.6 billion TiB of storage.

No cloud vendor has a billion harddisks, that's for sure.

(The filesystem might only support a maximum file size of 2^64 bytes, but that's nothing to do with the capacity of the storage array)

Re:ZFS 128 bit storage: Are you high? (2004) by wyoung2 on Monday February 19, 2018 @10:58AM (#16413608)
The article isn't talking about the size of raw disk blocks, better called the sector size. It's talking about ZFS blocks, which are variable in size, defaulting to 128 kB: https://en.wikipedia.org/wiki/ZFS#Variable_block_sizes

ZFS aside, the 512 byte sector isn't even a hard limit at the storage medium level. Disks with 528 byte sectors — and other odd sizes — have been available in the enterprise world for a long time now: https://en.wikipedia.org/wiki/Hard_disk_drive#Market_segment...

HDDs and SSDs with 4 kB sectors dominate the consumer market now: https://en.wikipedia.org/wiki/Advanced_Format

Re:ZFS 128 bit storage: Are you high? (2004) by killjoywashere on Monday February 12, 2018 @08:16PM (#16363970)
Isn't there an XKCD about the limit of flash storage being 3 PB per gallon?
Re:ZFS 128 bit storage: Are you high? (2004) by mewse on Monday February 12, 2018 @08:43PM (#16364090)
I think you may be referring to an image from this What-If article, where the capacity of a gallon of MicroSD cards is estimated at 1.6PB per gallon:

https://what-if.xkcd.com/31/

Re:ZFS 128 bit storage: Are you high? (2004) by dimensi0nal on Monday February 12, 2018 @09:03PM (#16364185)
It's 12.8PB per gallon now.
Re:ZFS 128 bit storage: Are you high? (2004) by perl4ever on Monday February 12, 2018 @10:44PM (#16364575)
A gallon of flash cards is great, but you do have to figure out which one you want after they are delivered. I'm wondering whether there is such a thing as an automated flash card library. I know people have archives of tapes like that.
Re:ZFS 128 bit storage: Are you high? (2004) by Dylan16807 on Monday February 12, 2018 @11:37PM (#16364804)
The thing about flash is that it's cheap and easy to keep it all permanently online. You can get a stack of 2.5 inch SSDs for the same price as microsd cards, and plugging them into a bus takes less space than a tape robot.

When you're looking at the same price for two systems, and one is 100x-1000x faster at the cost of being 'merely' hundreds of TB per gallon instead of a few PB per gallon, nobody is going to bother engineering the latter just to save a few bucks on shipping.

Re:ZFS 128 bit storage: Are you high? (2004) by detaro on Tuesday February 13, 2018 @09:29AM (#16367330)
Tape is a lot better at keeping data while in storage though.
Re:ZFS 128 bit storage: Are you high? (2004) by puzzle on Tuesday February 13, 2018 @08:23AM (#16366910)
"You know you have a large storage system when you get paged at 1 AM because you only have a few petabytes of storage left." (2010)

https://cloud.google.com/files/storage_architecture_and_chal...

No, they're not a single filesystem, as the papers make clear here and there.

Re:ZFS 128 bit storage: Are you high? (2004) by KaiserPro on Tuesday February 13, 2018 @07:24AM (#16366574)
S3/glacier are not filesystems, they are object storage interfaces with labels...
Re:ZFS 128 bit storage: Are you high? (2004) by rootw0rm on Monday February 12, 2018 @07:47PM (#16363831)
I wonder what backblaze is up to these days...
Re:ZFS 128 bit storage: Are you high? (2004) by Bud on Monday February 12, 2018 @06:30PM (#16363377)
Moore's Law actually predicted a doubling every two years (for CPUs), not every year, and even the very optimistic Kryder's Law (often cited as the version of Moore's Law for storage) only claimed an increase of 40% per year. Actual progress has fallen far short of that. The rate for 2010-2014 was only 15%/year.

Rates were quicker back around 2000-2005, but that pace has fallen dramatically.

https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_proje...

Re:ZFS 128 bit storage: Are you high? (2004) by wickawic on Monday February 12, 2018 @08:48PM (#16364108)
I think it is reasonable to expect storage to jump dramatically at such a time that we begin storing “3D” data. Not that everything in VR will be voxels or anything, but even if the fractal dimension of our future data is 2.5 it would obliterate our current needs for storage. /speculation
Re:ZFS 128 bit storage: Are you high? (2004) by AstralStorm on Tuesday February 13, 2018 @02:53AM (#16365456)
Physics provides us with certain density of information in a volume and that is considering advanced quantum storage.

May I remind you that flash, DRAM and SRAM already are 3D structures.

Re:ZFS 128 bit storage: Are you high? (2004) by Sean1708 on Tuesday February 13, 2018 @05:32AM (#16366113)
I think they meant VR graphics data, not data stored on 3D things.
Re:ZFS 128 bit storage: Are you high? (2004) by gowld on Tuesday February 13, 2018 @04:18AM (#16365781)
40%/yr is 100%/2years.
Re:ZFS 128 bit storage: Are you high? (2004) by mixmastamyk on Monday February 12, 2018 @10:36PM (#16364532)
Wonder what the pace of ssd capacity is?
Re:ZFS 128 bit storage: Are you high? (2004) by sliken on Monday February 12, 2018 @06:03PM (#16363189)
Last I checked the ZFS 128 bit values were not byte offsets, but block numbers. So the justification requires over 2^64 blocks, not 2^64 bytes.

I calculated it at some point and it was several years of the entire earths manufacturing capacity for disks in a single filesystem.

[added the missed "several years of"]

Re:ZFS 128 bit storage: Are you high? (2004) by dijit on Monday February 12, 2018 @06:08PM (#16363219)
If you accept the default 128k ZFS block size then it's 2,361,183,241,434,822,606,848 bytes, or 2,097,152 PB

Or, exactly 2 Zettabytes.

Re:ZFS 128 bit storage: Are you high? (2004) by Rapzid on Monday February 12, 2018 @06:18PM (#16363293)
Also, the Z in ZFS :D
Re:ZFS 128 bit storage: Are you high? (2004) by erric on Monday February 12, 2018 @07:12PM (#16363632)
>The ZFS name is registered as a trademark of Oracle Corporation; although it was briefly given the retrofitted expanded name "Zettabyte File System", it is no longer considered an initialism.

—Wikipedia

Re:ZFS 128 bit storage: Are you high? (2004) by bringtheaction on Monday February 12, 2018 @07:28PM (#16363718)
I don't believe Wikipedia is right. Search site:oracle.com for zettabyte on google and observe the results being documents and blogs from a multitude of years.

ZFS being an acronym for Zettabyte File System was neither retrofitted nor all that brief in use it seems.

Former employees of Sun Microsystems, I know some of you browse HN. Care to shed some light on this?

Re:ZFS 128 bit storage: Are you high? (2004) by bcantrill on Tuesday February 13, 2018 @03:29AM (#16365594)
I wasn't in the room when it happened, but I was probably among the first dozen or so to hear the name -- and it was presented to me as both at once: "ZFS" was both "the last word in filesystems" and (conveniently) a zettabyte. I remember Bonwick quickly pointing out that yes, a yottabyte was larger but "YFS" was passed over for obvious reasons.
Re:ZFS 128 bit storage: Are you high? (2004) by avar on Tuesday February 13, 2018 @03:45AM (#16365661)
What's that obvious reason?
Re:ZFS 128 bit storage: Are you high? (2004) by dijit on Tuesday February 13, 2018 @04:00AM (#16365719)
"Why? filesystem" I imagine.
Re:ZFS 128 bit storage: Are you high? (2004) by wyoung2 on Monday February 19, 2018 @11:09AM (#16413664)
Because "Z" matches the goal of being "the last word in filesystems" than "Y".
Re:ZFS 128 bit storage: Are you high? (2004) by Rapzid on Monday February 12, 2018 @07:34PM (#16363746)
Indeed, LLNL have a humongo Lustre/ZFS cluster and kickstarted the ZFSoL project(mucho gracias to LLNL and Brian Behlendorf) and they refer to it as a Zettabyte File System.
Re:ZFS 128 bit storage: Are you high? (2004) by krylon on Tuesday February 13, 2018 @07:37AM (#16366648)
Interestingly, IBM has a filesystem used on the zOS mainframe OS, that is also called ZFS: https://en.wikipedia.org/wiki/ZFS_(z/OS_file_system)
Re:ZFS 128 bit storage: Are you high? (2004) by marzell on Monday February 12, 2018 @06:36PM (#16363412)
So, if they didn't use 128-bit... they'd have to change the name! /s
Re:ZFS 128 bit storage: Are you high? (2004) by madaxe_again on Monday February 12, 2018 @06:37PM (#16363415)
CERN are at >100PB, which is only about 6.6 doublings, which is about right with a doubling every two, not one, years. http://iopscience.iop.org/article/10.1088/1742-6596/664/4/04...
Re:ZFS 128 bit storage: Are you high? (2004) by sliken on Monday February 12, 2018 @10:56PM (#16364624)
Right, but does CERN connect all 100PB to a single machine?
Re:ZFS 128 bit storage: Are you high? (2004) by tempay on Tuesday February 13, 2018 @03:58AM (#16365710)
No. Data is mostly accessed using XRootD[1] which can be used to data from an (almost) unlimited number of nodes. So far as I know, it's optimised for large files so the number of available inodes is limited by RAM.

There is also EOS[2] which is based on XRootD and is planned to replace AFS[3] as a general purpose networked filesystem.

[1] http://xrootd.org/ [2] http://eos.web.cern.ch/ [3] http://information-technology.web.cern.ch/services/afs-servi...

Re:ZFS 128 bit storage: Are you high? (2004) by AstralStorm on Tuesday February 13, 2018 @02:54AM (#16365459)
To a cluster fs which ZFS is not. (And not all of it.)
Re:ZFS 128 bit storage: Are you high? (2004) by ohazi on Monday February 12, 2018 @08:50PM (#16364125)
It was probably incorrect for the original author to cite Moore's law.

Moore's law only ever really held for transistor density on integrated circuits, not magnetic storage. Magnetic storage capacity always grew more slowly.

Flash storage capacity today has more to do with die stacking than transistor density.

In any case, Moore's Law is definitely dead now. Like, for real this time.

Re:ZFS 128 bit storage: Are you high? (2004) by runako on Monday February 12, 2018 @06:13PM (#16363259)
Backblaze posts some good stuff on HD pricing and reliability. Here's a good chart from their data:

https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/

They also link to this great (logarithmic) chart:

http://www.mkomo.com/cost-per-gigabyte-update

TL;DR: price hasn't been dropping by half annually, but prices have come down from ~$0.70/GB to < $0.03/GB now. So roughly a factor of 20 over 13-14 years, somewhere in the neighborhood of 25% improvements compounded annually over that time.

Re:ZFS 128 bit storage: Are you high? (2004) by ChuckMcM on Monday February 12, 2018 @06:26PM (#16363348)
> So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year.

Its unlikely. There are few opportunities to compile a data set of that size. One might be the NSA collections center in Utah, one might be Google or Microsoft's web cache, and one might be something like the Internet Archive's cache.

However the length choice gets 'weird' if you want something larger than 64 bits and less than 128. Your next intermediate choice is 96 bits. That is 64 + 32 (or one additional 32 bit long word. Growing by just 8 means you now have a 9 byte pointer, by 16 gives you a 10 byte pointer. Some architectures are penalized when doing off word alignment, and so on those architectures the structure is padded out to the next word size anyway.

I've always found Jef's reasoning in that post amusing but I continue to thing 64 bits would have been fine (just like I think it would have been fine for the Internet v6 work :-). Fortunately time has given us a crap ton of memory for our boxes so the storage penalty isn't too great.

Re:ZFS 128 bit storage: Are you high? (2004) by mjevans on Monday February 12, 2018 @06:48PM (#16363487)
Since 64 bits can serve /many/ use cases and also matches modern machine words I could see there being either two drivers (one with 64 bit native words and the other with full sized ones) or even a fancier driver that uses different ASM bits to just leave half the word all 0s and ignored.

The on-disk format could even remain the same; it wouldn't really matter except for lots of small files and those are already so horrid that ZFS might have some other way of handling them. (Like treating /all/ small file names and data as part of a larger directory file or something.)

Re:ZFS 128 bit storage: Are you high? (2004) by Johnny555 on Monday February 12, 2018 @09:36PM (#16364318)
Backblaze says they are using 400 PB of storage[1], or around 2^59 bytes. Of course, it's not one big filesystem. They aren't quite doubling annually, but looks like they are growing around ~1.5X each year, so in another decade, they could be close to the 2^64 limit.

[1] https://www.backblaze.com/company/about.html

Re:ZFS 128 bit storage: Are you high? (2004) by sliken on Monday February 12, 2018 @10:54PM (#16364615)
Right, but their building block is 48 disks x whatever is best profitable (a mix of 4,6,8,10, and 12TB currently or similar). So each system is 500TB or less, zfs is a filesystem for a single node not a distributed filesystem.

Even if we ignore that factor of 800 or so (400PB spread across 500 nodes), they still would need another factor of 128,000 before they would need ZFS's 65th bit.

Keep in mind it's 2^128 blocks, not 2^128 bytes.

Backblaze would need to consume the entire planets supply of disks for a very long time AND put all those disks connected to a single linux, freebsd, or solaris box. Only then would they need the 65th bit that ZFS has for addressing blocks.

Re:ZFS 128 bit storage: Are you high? (2004) by ryacko on Tuesday February 13, 2018 @03:29AM (#16365592)
Not really. Global hard drive production is more than 2^64 bytes or 18 exabytes (SI).

https://www.anandtech.com/show/10098/market-views-2015-hard-...

Re:ZFS 128 bit storage: Are you high? (2004) by tempay on Tuesday February 13, 2018 @03:43AM (#16365657)
From that it looks like production is 130 exabytes for Seagate and Western Digital combined. Using the default block size on ZFS the limit is 128 * 1024 * 2^64 = 2 zettabytes. Even allowing for other manufacturers it's an order of magnitude more than global annual production.
Re:ZFS 128 bit storage: Are you high? (2004) by jchw on Monday February 12, 2018 @06:05PM (#16363202)
I don't think Moore's Law held up for much since then, but it would've been hard to know for sure. That being said, ZFS filesystems can span multiple disks, since it does volume management of its own. I don't know if that means we're going to see zettabyte volumes any time soon, though. After all, 2^64 bytes wouldn't be enough to overflow: it would need to be 2^64 blocks...
Re:ZFS 128 bit storage: Are you high? (2004) by pasbesoin on Monday February 12, 2018 @11:48PM (#16364840)
An apparently overlooked aspect of the SpaceX Falcon Heavy launch, is the inclusion of a small quartz disc whose format is expected to scale to something like 260 TB (360?, I forget). And that has a lifespan claimed to potentially extend into the billion plus year range.

I don't know about that exact disc/format, but if we and as we are finally able to stably write at such capacities... (whatever technologies end up being used to do so).

We're going to run into another "640KB should be enough for anyone", unless we are forward thinking with regard to potential storage format capacities.

(And if you think no one will ever be able to consume that much data and detail, just think of all the modeling that will be done and the data behind that. Think of the data being put out by CERN and not just how it will expand but also how people will want to explore lower significance hints, corner cases, and who knows what. Etc.)

Re:ZFS 128 bit storage: Are you high? (2004) by anardil on Tuesday February 13, 2018 @12:02AM (#16364888)
A fully decked out Isilon cluster supports 68PB on a single file system.

https://www.emc.com/collateral/hardware/white-papers/h10719-...

Re:ZFS 128 bit storage: Are you high? (2004) by wjkohnen on Tuesday February 13, 2018 @01:32PM (#16369506)
ZFS is not really a 128 bit filesystem, it is merely a 64 bit filesystem that is "128 bit ready". I.e. the on disk format is 128 bit, the kernel drivers are not. At least, they were not until Oracle bought Sun, that's about the last time I really cared about ZFS.