I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here.
Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".
This isn't a random programmer philosophising about POSIX, it's an ad for a product.
Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().
What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.
Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone
I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)
New != Good
Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.
So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
https://github.com/s3fs-fuse/s3fs-fuse
Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).
Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?
There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).
it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs
I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?
Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.
Could the POSIX standard shell move that far? Or is there absolutely no hope?
Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that
https://lwn.net/Articles/343924/
And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.
At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.
"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.
"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."
Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.
This is why POSIX Powershell could never exist.
https://news.slashdot.org/story/01/02/06/2030205/david-korn-...
That's completely uncalled for and obnoxious.
1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems).
2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.
S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions.
But ultimately, they're both storing files. The differences are fairly minor.
So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).
https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.
After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure.
So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications.
(not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance)
We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful.
> The sort of cloud storage systems described all run on top of POSIX APIs
Is that a benefit or a cost?
I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.
Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different.
"I want object storage APIs for local storage" <= completely reasonable
"I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable
Even if you build your scalable production environment from scratch with native object storage support (which is rare!), it’s eminently useful to be able to use coreutils to interact with said object storage for devops: nothing beats e.g. running grep on a list of log “files” in an S3 bucket in terms of convenience.
Hell, why do so many static pages even need javascript
Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.
You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance.
And yet, it's still a very useful thing to do.
The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful.
Hell, the extistence of terraform should make that loud and clear.
POSIX is absolutely outdated, but this article isn't helpful on understanding why or how.
Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
There's about as much reason to expect a single API to access the filesystem attached to the PCI bus and an unstructured storage system as there is to have a single API to create a PDF of a document AND deliver it to the printer. Which is to say: the higher level abstraction will use the lower level anyway, but it exists for a reason, and isn't appropriate for everything.