News for hackers. Stuff that matters.

cuno

Re:Is POSIX really outdated? on Thursday October 19, 2023 @11:06AM (#37943903)
Hi, author here.

Yes, we've had to do a lot of things to deal with object storage latency. Since cunoFS is running inside a given process itself, it has greater visibility into what actions it can take and is likely to take. This means we can make huge improvements in our prediction logic, so that we can prefetch within and across files much better, thus hiding latencies. For POSIX metadata, we have a caching mechanism which is shared between processes, and we've invented a better way to encode this metadata so that it is retrieved alongside the LIST operation.

Re:Is POSIX really outdated? on Thursday October 19, 2023 @10:52AM (#37943707)
Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.
Re:Is POSIX really outdated? on Thursday October 19, 2023 @10:37AM (#37943500)
Hi, author here.

The problem is that it's actually really hard to write you own specialized glue layer,and so most developers who do that have poor implementations with low performance and/or incompatibilities with non-AWS solutions. In the context of object storage, we've found lots of applications have tried to add S3 support, and while they work with AWS S3 and have basic functionality, they fail on a lot of other S3-compatible solutions. So they end up tied to AWS, and you can't use them on Microsoft Azure Storage, or often on Google Cloud Storage (despite its S3 gateway), or others.

For instance, a key workhorse of the genomics field is `samtools`, which works with AWS S3 in some ways, but not others (like Amazon Resource Names[1]). Our approach works across vendors transparently, and on S3, is much faster compared to such native implementations.

[1]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-a...

Re:Is POSIX really outdated? on Thursday October 19, 2023 @10:08AM (#37943141)
Hi, author here.

Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().

Re:Is POSIX really outdated? on Thursday October 19, 2023 @10:07AM (#37943130)
Hi, author here.

Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).

Re:The next century of computing on Tuesday September 27, 2022 @05:59AM (#32993384)
Some of these predictions are already happening so maybe "prediction" is the wrong word? Traditional computation has for a long time assumed that "computation" is expensive and "communication" between logic components is almost free. That's reversed now - the communication needs of computation are already a lot more expensive than the raw computation needs (transistors/gates).

That means something like binary tree traversal can no longer be O(logN) it actually becomes O(sqrt(N)), or at best O(N^(1/3)). We're also seeing that hardware circuits, computer software and the human brain already have common emergent communication behaviour. If you're interested in fundamentals of what this means for future computation see for example:

Communication Locality in Computation: Software, Chip Multiprocessors and Brains https://www.cl.cam.ac.uk/~swm11/research/greenfield.html