News for hackers. Stuff that matters.
Re:Is POSIX really outdated? by PaulDavisThe1st on Thursday October 19, 2023 @09:25AM (#37942561)
Sigh. Another article written, it seems, by someone who thinks that all/most software development is on the web. "Storage systems" in the cloud ... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. But this is an illusion likely caused by insufficient native development experierence.

The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

There's about as much reason to expect a single API to access the filesystem attached to the PCI bus and an unstructured storage system as there is to have a single API to create a PDF of a document AND deliver it to the printer. Which is to say: the higher level abstraction will use the lower level anyway, but it exists for a reason, and isn't appropriate for everything.

Re:Is POSIX really outdated? by VyseofArcadia on Thursday October 19, 2023 @09:35AM (#37942691)
Thank you for phrasing this so much more politely than I would have.

I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here.

Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".

Re:Is POSIX really outdated? by jeroenhd on Thursday October 19, 2023 @09:45AM (#37942827)
"POSIX file system for the web" is kind of the whole deal for the company that wrote this. They make something to mount object storage as a filesystem.

This isn't a random programmer philosophising about POSIX, it's an ad for a product.

Re:Is POSIX really outdated? by cuno on Thursday October 19, 2023 @10:08AM (#37943141)
Hi, author here.

Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().

Re:Is POSIX really outdated? by rewmie on Thursday October 19, 2023 @07:32PM (#37950217)
> (...) leverage the lower cost and higher throughput of object storage like S3 (...)

What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.

Re:Is POSIX really outdated? by mcv on Friday October 20, 2023 @09:10AM (#37955514)
Not to mention that apparently we're supposed to compare that to storing on your local harddisk. Isn't local storage always far higher throughput than accessing something across the internet? Being able to store and access stuff on the internet as if it's local is a great idea, but the way they sell it is unconvincing and sounds like they don't really understand what they're talking about.
Re:Is POSIX really outdated? by cuno on Friday October 20, 2023 @06:05PM (#37962025)
Hi author here, sorry I missed this post. The performance benchmarks and cost comparisons are for comparing S3 vs EBS (ext4 formatted), EFS, FSx Lustre and others within the same datacenter (i.e. LAN use case rather than WAN use case). That means if you have an EC2 instance running in, say AWS Ohio, and are comparing those storage options also within AWS Ohio, then cunoFS is both cheaper and higher throughput than those other options. It's a different story over WAN. In that case, your own local NVMe storage is going to be cheaper and generally faster that remote storage over a WAN. But that local NVMe storage (on say your solo laptop) isn't going to have anywhere near the Enterprise-grade redundancy, availability and scalability that AWS S3/Azure Blob/Storj/Wasabi/etc has.
Re:Is POSIX really outdated? by pdntspa on Thursday October 19, 2023 @10:34AM (#37943468)
It smells like a bunch of FUD to me

Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone

Re:Is POSIX really outdated? by psd1 on Friday October 20, 2023 @06:37AM (#37954421)
FUD is common both ways. Your comment could be interpreted as FUD against anything new.

I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)

Re:Is POSIX really outdated? by pdntspa on Friday October 20, 2023 @08:29PM (#37962990)
Yeah, I use all of those. But what I can't really use is some low-rent sales copy trying to persuade me to abandon FOSS tooling in favor of yet another rent-seeking cloud solution.

New != Good

Re:Is POSIX really outdated? by psd1 on Sunday October 22, 2023 @01:10PM (#37977144)
Agreed, with only one slight quibble: content can be good even when the purpose is sales.
Re:Is POSIX really outdated? by jasode on Thursday October 19, 2023 @09:45AM (#37942829)
>... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. [...] The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.

So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

https://github.com/s3fs-fuse/s3fs-fuse

https://github.com/kahing/goofys

https://www.cuno.io/

Re:Is POSIX really outdated? by cuno on Thursday October 19, 2023 @10:07AM (#37943130)
Hi, author here.

Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).

Re:Is POSIX really outdated? by jonhohle on Thursday October 19, 2023 @10:42AM (#37943587)
> in fact, in most cases, there is no filesystem at all

Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?

There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).

Re:Is POSIX really outdated? by cuno on Thursday October 19, 2023 @10:52AM (#37943707)
Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.
Re:Is POSIX really outdated? by IcePic on Thursday October 19, 2023 @10:54AM (#37943730)
CEPH would be one of those, which is more like a database using raw devices, and no linux filesystem (nor VFS calls) in between. It used to run on top of xfs, but now storage nodes (OSDs) talks directly to a raw device (or an lvm on top of it). But you can most certainly not run "ls" on such a host and get a listing of what some user wrote over S3 into it.
Re:Is POSIX really outdated? by jonhohle on Thursday October 19, 2023 @12:14PM (#37944839)
Are there any benchmarks that justify bypassing VFS? My Google-fu is failing and the one page I found comparing file systems in the CEPH docs doesn’t talk about raw access, all images are broken (stored on CEPH raw storage?), and the text doesn’t seem to include throughout numbers.
Re:Is POSIX really outdated? by tryauuum on Thursday October 19, 2023 @12:31PM (#37945049)
https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf my google-fu found this

it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs

Re:Is POSIX really outdated? by IcePic on Thursday October 19, 2023 @01:57PM (#37946194)
If you store stuff on xfs or ext4, it will cause more writes due to the journal, and since ceph is a DB in some sense, it also keeps a journal/WAL so doubling that is a waste.
Re:Is POSIX really outdated? by LightFog on Friday October 20, 2023 @12:37AM (#37952433)
It’s not uncommon for high performance object stores to bypass POSIX FS semantics and provide kernel patches etc - eg lustre do this.
Re:Is POSIX really outdated? by chasil on Thursday October 19, 2023 @11:02AM (#37943857)
One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.

I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?

Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.

Could the POSIX standard shell move that far? Or is there absolutely no hope?

https://frippery.org/busybox/index.html

Re:Is POSIX really outdated? by ilyt on Thursday October 19, 2023 @11:39AM (#37944398)
> One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.

Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that

https://lwn.net/Articles/343924/

And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.

At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.

Re:Is POSIX really outdated? by chasil on Thursday October 19, 2023 @11:46AM (#37944490)
This is very much not true.

"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.

"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."

Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.

This is why POSIX Powershell could never exist.

https://news.slashdot.org/story/01/02/06/2030205/david-korn-...

Re:Is POSIX really outdated? by chasil on Thursday October 19, 2023 @03:16PM (#37947351)
Interesting how expressed frustration with POSIX.2 is received.
Re:Is POSIX really outdated? by biorach on Thursday October 19, 2023 @12:11PM (#37944797)
> So please don't spew bullshit reasons you think something happened as gospel.

That's completely uncalled for and obnoxious.

Re:Is POSIX really outdated? by nickik on Thursday October 19, 2023 @01:08PM (#37945538)
ZFS new object store backend give you that interface I think.
Re:Is POSIX really outdated? by calvinmorrison on Thursday October 19, 2023 @09:54AM (#37942937)
APIs keep getting conflate for REST APIs. As far as I understand it, the stdlib is an api
Re:Is POSIX really outdated? by nvm0n2 on Thursday October 19, 2023 @10:00AM (#37943043)
I don't know. Consider:

1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems).

2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.

S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions.

But ultimately, they're both storing files. The differences are fairly minor.

Re:Is POSIX really outdated? by PaulDavisThe1st on Thursday October 19, 2023 @12:47PM (#37945251)
> By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel.

So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).

Re:Is POSIX really outdated? by yencabulator on Thursday October 19, 2023 @01:40PM (#37945964)
[delayed]
Re:Is POSIX really outdated? by erik_seaberg on Thursday October 19, 2023 @03:55PM (#37947865)
Originally not, but they advertise strong read-after-write consistency as of December 2020.

https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

Re:Is POSIX really outdated? by yencabulator on Thursday October 19, 2023 @04:12PM (#37948080)
As far as I know, you still don't get e.g. create-if-not-exist. The object listing is now strongly consistent, but you don't get to do atomic operations on it.
Re:Is POSIX really outdated? by pjmlp on Thursday October 19, 2023 @10:07AM (#37943132)
Except many of those systems are implemented on top of type 1 hypervisors, running either containers or minimal runtimes, and if there is any POSIX related APIs, it is only the minimal surface as required by ISO C and ISO C++ standards.

POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.

Re:Is POSIX really outdated? by pjc50 on Thursday October 19, 2023 @10:09AM (#37943155)
As a much more low-level developer, I think they have a point, but not that the interface should be exactly S3.

After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure.

So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications.

(not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance)

We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful.

> The sort of cloud storage systems described all run on top of POSIX APIs

Is that a benefit or a cost?

Re:Is POSIX really outdated? by PaulDavisThe1st on Thursday October 19, 2023 @12:49PM (#37945276)
I'm not disputing the idea that sometimes (much) higher level abstractions in an API are a good idea.

I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.

Re:Is POSIX really outdated? by delta_p_delta_x on Thursday October 19, 2023 @10:39AM (#37943533)
[delayed]
Re:Is POSIX really outdated? by PaulDavisThe1st on Thursday October 19, 2023 @12:52PM (#37945309)
> There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns

Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different.

"I want object storage APIs for local storage" <= completely reasonable

"I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable

Re:Is POSIX really outdated? by klmr on Friday October 20, 2023 @06:09AM (#37954252)
Why? Because all typical, existing applications written for POSIX systems use the POSIX API (usually indirectly) to interact with the filesystem and perform IO. Being able to use that vast ecosystem of existing applications seamlessly with object storage is such a common requirement that a good dozen different object storage-to-POSIX abstraction layers exist (many of them on top of FUSE).

Even if you build your scalable production environment from scratch with native object storage support (which is rare!), it’s eminently useful to be able to use coreutils to interact with said object storage for devops: nothing beats e.g. running grep on a list of log “files” in an S3 bucket in terms of convenience.

Re:Is POSIX really outdated? by jra_samba on Thursday October 19, 2023 @07:47PM (#37950357)
Have you tried running Samba on top of cunoFS ? I'd be interested in performance data on that. For SMB3 access it might be better to add a Samba VFS vfs_cunoFS interface directly. You won't need to open source your client code, just the wrapper.
Re:Is POSIX really outdated? by cuno on Friday October 20, 2023 @06:10PM (#37962058)
Hi sorry I missed this. Yes, you can run Samba for SMB, and Ganesha for NFSv4. Feel free to email us if you think there's a better way though.
Re:Is POSIX really outdated? by jra_samba on Monday October 23, 2023 @06:30PM (#37992498)
Yes, I think there's a better way (which would make Samba faster on your system :-). Can you ping me at [email protected] and I'll explain !
Re:Is POSIX really outdated? by ChrisRR on Thursday October 19, 2023 @11:17AM (#37944051)
If anything web dev is way worse than the POSIX standard as everything changes every 6-12 months.

Hell, why do so many static pages even need javascript

Re:Is POSIX really outdated? by pjmlp on Thursday October 19, 2023 @11:59AM (#37944660)
Because obviously we need to generate them from Next.js. /s
Re:Is POSIX really outdated? by kazinator on Thursday October 19, 2023 @12:13PM (#37944831)
It was produced by someone thinking, "why don't I ask AI what is the difference between cloud storage APIs and POSIX".
Re:Is POSIX really outdated? by OJFord on Thursday October 19, 2023 @12:16PM (#37944860)
It reads to me more like the title is 'flamebait' if you like, and you've been baited by it; what the article actually seems, to me, to be saying is 'POSIX good, but many things use S3-like object storage, we need a POSIX-like standard there, by the way we make cunoFS a POSIX-compat layer for "S3-compatible"'.

Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.

Re:Is POSIX really outdated? by PH95VuimJjqBqy on Thursday October 19, 2023 @01:28PM (#37945805)
fuse is a thing and it suffers from the exact same problem.

You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance.

And yet, it's still a very useful thing to do.

The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful.

Hell, the extistence of terraform should make that loud and clear.

Re:Is POSIX really outdated? by marcosdumay on Thursday October 19, 2023 @01:37PM (#37945917)
Yet, the largest issues with networked filesystems is that POSIX forbids canceling file accesses, the lock semantics is really difficult, and cache is hard.

POSIX is absolutely outdated, but this article isn't helpful on understanding why or how.

Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.