str4d

> The XDG specification has been around for 22 years.

Indeed!

> It has real benefits for users.

I once believed this!

> It's trivial to implement.

Nope!

I ranted about this a while back (https://bsky.app/profile/str4d.xyz/post/3lsjbnpsbh22i) but the tl;dr is that the XDG specification is imprecisely written and internally inconsistent, and there has been confusion since at least as early as 2003 about what kind of data is meant to be stored in $XDG_CONFIG_HOME vs $XDG_DATA_HOME.

A plain reading of the XDG Base Directory Specification implies (but does not explicitly say, which is part of the problem!) that $XDG_DATA_HOME contains read-only user-specific application data files (like documentation).

But the spec is interpreted by many (source: I found a single blog post that seemed to be referenced in a variety of places, such as the Arch Linux wiki) as saying that $XDG_DATA_HOME contains mutable user-generated data (like password vaults).

Those have very different properties, and it's impossible as an app developer for me to predict which of those two directories a user (or the tooling installed on their OS) is assuming can be safely deleted without consequence, and which must always be backed up.

> Yet even in the year of our lord two thousand and twenty five I still see TypeScript developers complain that it's "too hard" to comply with "this BS" and just stick files in $HOME.

My earlier rant arose from me spending several days attempting to follow the XDG specification for an app where I need to store mutable user-generated data, being unable to find any supporting evidence that it was safe to store this in either $XDG_CONFIG_HOME or $XDG_DATA_HOME, and deciding on ~/.appname instead. I do allow all paths to be overridden in the config file, so individual users can use XDG paths at their own risk (and with knowledge of their OS environment's behaviour).

> The firehose is all public data going into the network right?

It's the "main subset" of the public data, being "events on the network": for the Bluesky app that's posts, reposts, replies, likes, etc. Most of those records are just metadata (e.g. a repost record references the post being reposted, rather than embedding it). Post / reply records include the post text (limited to 300 graphemes).

In particular, the firehose traffic does _not_ include images or videos; it only includes references to "blob"s. Clients separately fetch those blobs from the PDSs (or from a CDN caching the data).

> > If you want large scale social networks, you need to work with a large scale of data. Since federated open queries aren’t feasible, you need big machines.

> Thats just simply not true.

> [snip]

> This is why ActivityPub has a discovery problem by the way, but it's just a symptom of real federation.

You're actually agreeing with them! The "discovery problem" is because "federated open queries aren't feasible".

> With ATProto you NEED to connect to some centralized relay that will broker your messages for you.

You can connect to PDSs directly to fetch data if you want; this is exactly what the relays do!

If you want to build a client that behaves more like ActivityPub instances, and does not depend on a relay, you could do so:

- Run your own PDS locally, hosting your repo.

- Your client reads your repo via your PDS to see the accounts you follow.

- Your client looks up the PDSs of those accounts (which are listed in their DID documents).

- Your client connects to those PDSs, fetches data from them, builds a feed locally, and displays it to you.

This is approximately a pull-based version of ActivityPub. It would have the same scaling properties as ActivityPub (in fact better, as you only fetch what you need, rather than being pushed whatever the origins think you need). It would also suffer from the same discovery problem as ActivityPub (you only see what the accounts you follow post).

At that point, you would not be consuming any of the _output_ of a relay. You would still want relays to connect to your PDS to pull data into their _input_ in order for other users to see your posts, but that's because those users have chosen to get their data via a relay (to get around the discovery problem). Other users could instead use the same code you're using, and themselves fetch data directly from your PDS without a relay, if they wanted to suffer from the discovery problem in exchange for not depending on a relay.

Additionally, a user _can_ root their identity in DNS if they want, by using did:web instead of did:plc [0]. The main Bluesky client doesn't expose this (presumably because did:web cannot provide a mechanism for automatic migration between PDSs (due to the PDS having no control over the DID document) or recovering from loss of control of the domain name, so it requires more technical expertise), but there are users successfully using this method for their identities.

[0] https://atproto.com/specs/did#blessed-did-methods

For a lower-friction firehose experience, you can use Jetstream [0] (by one of the Bluesky devs) which supports subscribing to specific Collection NSIDs and user repositories, and converts records to JSON for you.

There's a public instance URL in the README (with bandwidth limits), or you can self-host.

[0] https://github.com/ericvolp12/jetstream

For non-Bluesky apps built in ATProto, in addition to White Wind (blogging), there is also Smoke Signal (events, only Lexicons are open source currently AFAICT) [0], and Frontpage (link aggregation) [1].

[0]: https://github.com/SmokeSignal-Events/lexicon

[1]: https://github.com/likeandscribe/unravel/tree/main/packages/...

There are a few more details about the reason they didn't use JSON-LD in Paul's blog post [0].

[0]: https://www.pfrazee.com/blog/why-not-rdf

> But I’m still wondering how one would aggregate reactions. It seems it would not be enough just to have a PDS, but also to run an Indexer/Relay, which indexes other peoples PDS. And not just the PDSs of the people you are following, but also the rest of the world – after all a Like or such can come from everyone. When running an Indexer you are pretty soon into Firehose territory. Too much for a single person website.

You don't need to run a Relay yourself in order to see reactions; you instead subscribe to an existing Relay's firehose (such as the one that BlueSky PBC operates), and ignore anything that isn't "reaction-shaped" (matches the relevant Lexicon).

> Everyone talking about ATP seems to [..] never talk about the transmission mechanisms. Whereas I’m sitting here and thinking that an ActivityPub Actor’s Outbox is already a form of PDS – and it has an Inbox. Addressable via the Web and findable via Webfinger.

Here's my attempt to summarise the transmission difference between ActivityPub and AT Protocol: ActivityPub is symmetric and push-based, AT Protocol is asymmetric and pull-based.

With ActivityPub, your client talks to your server. The server handles both outgoing data (you making posts) and incoming data (viewing other people's posts).

When you create a post:

- The post is placed in your server's outbox.

- Your server looks up the servers for every one of your followers, and places your post in their inbox.

When someone creates a reaction:

- The reaction is placed in their server's outbox.

- Their server looks up your server, and places the reaction in its Inbox.

- Your website checks the Inbox, discovers the reaction, and handles it.

With AT Protocol, your client talks to your PDS and the client's App View. Your PDS handles outgoing data (you making posts), and your client's App View handles incoming data (viewing other people's posts). Relays are what provide transmission between the two: Relays subscribe to your PDS, and App Views subscribe to the firehose of at least on Relay.

When you create a post:

- The post is added to your repository on your PDS.

- The Relay pulls the post from your PDS and broadcasts it on its firehose.

- App Views see the post in the firehose, and include it within the feeds of users that follow you.

When someone creates a reaction:

- The reaction is added to their repository on their PDS.

- The Relay pulls the reaction from their PDS and broadcasts it on its firehose.

- Your website's App View (which might be your client's App View if "reactions" are also posts, or some other App View if this is a different kind of semantic data) sees the reaction in the firehose, and handles it (e.g. includes it within the feed of reactions that your website uses).

A useful way to think about it is that the PDS is a "user agent" - they act on behalf of the user, and with the user's implicit trust. This is much the same way that a user trusts webserver software (and the VPS running it) to correctly serve their website, hold onto the private keys for the site's TLS certificates, use those private keys to correctly set up encrypted connections to the website, etc.

The AT Protocol itself does technically allow for all private key material to be held at all times by the user, but that means the user needs to be interactively involved in any operation that requires making a signature (and online / available whenever such an operation is necessary, which isn't necessarily the same times as when the user is interacting with a particular app like BlueSky running on ATProto). The PDS software that BlueSky have implemented instead requires (IIRC) that the PDS has the private key for at least one of the public signing keys (but there can be multiple signing keys, so the user can also hold onto one locally).

> The indexer is only necessary because the user ids do not remain static on the system. Instead there is some weird cryptographic reshuffling going on where DID:PLC starts as a substring of a hash and then because of 'key rotation' the user id of every user actually changes out from under the user themselves over time.

The user IDs (e.g. did:plc or did:web) do not change over time. You are correct that a user that uses did:plc has that generated from a hash (of their initial repository state), but subsequent additions to or deletions from the repo don't change the DID. Likewise, if you set up your repository with did:web, the domain you pick is fixed, and you lose access to your repo if you lose control of that domain.

What is changing with key rotation is the repo signing keys that authorize updates to the user repo. If you run an entirely self-hosted PDS then you can choose to never rotate that key, and then you have a "static private key" repository. But key rotation is a necessary part of the design, in part to enable account migration (you wouldn't want an old PDS to still be allowed to make changes to your repository, so you'd rotate those keys to remove the old server's key and add the new server's key).

> There is a sub-curve (Ristretto) which is of prime order which can be used on top of curve25519.

I'm sure you are aware of this and are just abbreviating, but I think it's important to clarify this for other readers: Ristretto255 is not a sub-curve of Curve25519. It is not a curve at all. Ristretto is abstractly a prime-order group, and concretely a scheme for encoding group elements such that round-trip decode-and-encode produces the same element encoding.

It happens to have been designed to have the same order as the prime-order subgroup of Curve25519, for convenience of implementation, but it is possible to implement it using a different elliptic curve under the hood (though in general you're better to use Curve25519 under the hood for the same reason as avoiding Ed448: there are many more eyes working on solid implementations of that curve's arithmetic).

> But doing zero-knowledge proofs of Ristretto operations is significantly more complicated due to the fact that it is a sub-curve

Using Ristretto255 in zero-knowledge proofs is not much more complex than using an elliptic curve. It just isn't as _simple_ as using a plain elliptic curve, because you cannot witness a Ristretto element as a coordinate pair (x, y), again because Ristretto255 is not a curve. You instead have to witness the encoding of the group element, and then implement the decoding operation inside the circuit to constrain the encoding to its arithmetic representation (e.g. a Curve25519 curve point). This would actually be pretty straightforward to implement in a PLONK-style arithmetisation, but it's also only 12 field element multiplications and an inversion constraint, so it's perfectly feasible in R1CS as well. And thanks to the decoding constraints providing a firewall between the group element encoding and its arithmetic representation, once you have the latter you can safely just use it, and don't need to propagate subgroup checks or cofactor elimination throughout your protocol (like you would using Curve25519 directly).

In practice though, I agree that if you can use a prime-order elliptic curve, then you should definitely do so. Ristretto255 was designed at a time when people wanted to use cofactor curves for performance and curve-safety at the cost of group-safety; Ristretto255 provides that group-safety. Nowadays we have complete and performant formulae for all of the prime-order elliptic curves people want to use, so if you have the option then both secp256r1 and secp256k1 are decent options. But if you're in an ecosystem where Curve25519 is a necessary primitive, then Ristretto255 is a great option.

(Disclosure, since I'm not using a pseudonym here: Filippo and I are authors on the Ristretto IETF draft [0].)

[0] https://datatracker.ietf.org/doc/draft-irtf-cfrg-ristretto25...

I was briefly _very_ confused when I saw the news.

> But if I recall, it seems that the age key file itself is 128 bits! Thus in the symmetric mode, Age provides only 128 bits of security. This is substandard, and in particular secure only until 2035 or so.

This is not the case. To meet NIST's Category 1 requirements, post-quantum cryptography "must require computational resources comparable to or greater than those required for key search on a block cipher with a 128-bit key (e.g. AES128)" [0]. It is therefore completely appropriate to pair any Category 1 post-quantum KEMs with 128-bit symmetric keys. See [1] for more details about post-quantum age. Symmetric passphrase age similarly is fine because it's the same as the post-quantum KEM case but without the KEM.

[0] https://csrc.nist.gov/projects/post-quantum-cryptography/pos...

[1] https://words.filippo.io/dispatches/post-quantum-age/

Ooh, nice! Having a Kotlin-native implementation will be a significant UX improvement over e.g. using my Rust age implementation via FFI / JNI behind a wrapper.

It also reminds me (again) of the cross-language plugin question. The CLI apps interact with plugins using an IPC protocol over stdin/stdout. The idea was that the plugin side of the IPC could be placed in front of a regular Recipient or Identity library interface, so plugin authors could offer both a plugin binary for any CLI app, and a plugin library that can be compiled into other apps of the same language. Libraries in other languages sit somewhere in the middle ^_^;; It's probable that e.g. my age-plugin-yubikey Rust code could also be wrapped via JNI and used directly as a library, but there's maybe an IPC-over-FFI approach that could work as well. It would definitely be nice if only the main age client needed to be ported to target languages (though it should be similarly possible for language ports of plugins to work).

> Tornado Cash is a service, _run by an individual_ who was warned to cease serving sanctioned entities, and failed to do so.

This is false. The (vast majority of the) Tornado Cash contracts were either deployed to Ethereum as immutable contracts, or updated in 2020 to revoke mutability (once the final zkSNARK parameters were included) [0], meaning that they could not later be updated by the user(s) that deployed them. Arguing that Tornado Cash is run by an individual means arguing that the entire Ethereum network is run by an individual. There was no way for a warned individual to comply with that warning.

[0] https://www.coincenter.org/education/advanced-topics/how-doe... - Section titled "Can Tornado Cash be removed or updated? If so, by whom?"

> A good feature of PIV applet of Yubikey 5 is that it stores 24 keys.

Note that not all 24 of those keys are suitable for age usage. The 4 main keys have specific usage definitions in the PIV specification that mean hardware tokens alter how those key slots behave. Only one of them (the KeyManagement slot) has a definition that allows encryption, and even that I was somewhat suspicious of overlapping with, as I couldn't predict how those existing keys were being used, and didn't want to support every possible key type that might be in that slot (which users likely wouldn't be able to alter).

age-plugin-yubikey avoids this complexity by only interacting with the 20 "retired" slots, which have no constraining definitions. (I am considering adding restricted support for the KeyManagement slot specifically for CAC card users who aren't allowed to add new keys to their cards [0], but this would be behind a default-off feature flag to keep the primary UX simple.)

[0] https://github.com/str4d/age-plugin-yubikey/issues/62

Yep! Plugins themselves are ephemeral - all of their runtime state is provided by the age client - but when the plugin binary is invoked by the age client, it can then connect to (or start) the long-lived agent process and act as a proxy to it.

For YubiKey support specifically, my age-plugin-yubikey plugin handles encryption and ephemeral decryption (meaning that it connects to the YubiKey live, and thus has to treat e.g. "Once" PIN policies as "Always"). Once something like yubikey-agent has been extended to provide an age plugin, you could then take an age-plugin-yubikey identity and convert it into an agent plugin identity (so that the age client knows to invoke the agent's plugin for decryption rather than age-plugin-yubikey).

> and also a mobile app that can do this (right now I use PasswordStore on Android, which communicates with OpenKeyChain which is basically GPG for Android with a nice shell around it). So I use the Yubikey over NFC there (though USB is also possible if you really want), to access my passwords. It works pretty well after some initial hiccups.

The age plugin system and its currently-specified IPC [0] focuses on easy interoperability via plugin discovery in the PATH. This works very well for native desktop applications (and even between e.g. a Windows host and WSL guest [1]), but obviously not for web or mobile. I'm not optimistic about dynamic plugin discovery ever being feasible for web, but in theory alternate plugin-discovery mechanisms could be figured out for mobile (e.g. Android Intents).

Fortunately, the age plugin system is _also_ a relatively thin wrapper around the core age concepts of Recipients and Identities, so it's possible to write code for an age plugin that can also be used directly as a library! I haven't tried this with age-plugin-yubikey yet specifically, but it should be possible (after some refactoring) to produce a library that can be used to statically include support for YubiKeys directly in an age mobile app, backed by whatever Android library is currently used to provide YubiKey PIV access.

[0] https://github.com/C2SP/C2SP/pull/5

[1] https://github.com/str4d/age-plugin-yubikey#windows-subsyste...

The change in hash function did also result in speed improvements. Quoting the commit messages:

Change to BLAKE2s:

> This also has the advantage of supplying 16 bytes at a time rather than SHA1's 10 bytes, which, in addition to having a faster compression function to begin with, means faster extraction in general. On an Intel i7-11850H, this commit makes initial seeding around 131% faster.

RDRAND call removal:

> Removing the call there improves performance on an i7-11850H by 370%. In other words, the vast majority of the work done by extract_crng() prior to this commit was devoted to fetching 32 bits of RDRAND.

My answer to that is "maintainability". The more flexibility and moving parts we add to the RNG, the more things that can break. The delta from "one hash function" to "two hash functions" would involve not just adding a second hash function, but also a bunch of configuration code, target-specific logic, fallback handling, etc. There are plenty of places in the kernel that need to care about this kind of logic, but I don't believe the RNG is one of them, and I don't particularly want to require the RNG maintainers to spend their time caring about it.

Additionally, it's not just the hash function that would matter for speed here, but also the expansion. Linux RNG uses ChaCha20 for that, so if you were going all-in on target-specific speed, you'd need additional logic for swapping that out for a hardware-accelerated cipher (probably AES, which would introduce even more considerations given that it has a 16-byte block size, vs ChaCha20's 64-byte blocks).

The same thing happens with age-plugin-yubikey. You can only generate identities on the device (so you never have access to the private key), and the YubiKey needs to be plugged in and available in order to decrypt, as the core asymmetric decryption operation occurs on the YubiKey. (You don't need the YubiKey to _encrypt_, just the recipient string).

The way this is set up in the plugin protocol, the age client provides the age header to the plugin, along with any identities the user has asked to decrypt with, and the plugin returns any file keys it is able to decrypt. The process of taking the per-recipient encrypted blobs in the header and extracting file keys, is entirely up to the plugin. This falls nicely out of the way the age format's "one joint" is designed: the file is encrypted with one symmetric key, and that key is encrypted separately to each recipient via whatever mechanism their type requires.

If you squint through the layers upon layers of packet complexity, this is the same way GPG encryption operates [0]: the data is symmetrically encrypted with a random session key, and that session key is asymmetrically encrypted to each recipient. Only the asymmetric decryption part happens on-device for GPG; the symmetric part happens on the computer.

[0] https://datatracker.ietf.org/doc/html/rfc4880#section-2.1

A few people have been experimenting with OS integration (e.g. Windows context menu), but IDK how far along those experiments are.

I've been experimenting with a web-based UI [0] for my Rust implementation of age, because I really wanted drag-and-drop but could not figure out how to persuade any Rust GUI libraries to do it ^_^;

[0] https://rage-encry.pt/

> o It's odd that -e (encrypt) can recognize an age-keygen file, but -d (decrypt) errors out with: > > age: error: failed to parse recipient file "person.pub": "person.pub": malformed recipient at line 1

I'm confused as to how you reached this error message; "age -d" doesn't support recipient files / -R, only identity files (which is what age-keygen produces). It would be helptul to open an issue showing how to reproduce; either there's a bug, or documentation could be improved.

> o Not available as a brew install

While age was in beta, it provided a brew tap. But now that 1.0.0 has been released, it has just (3 hours ago!) been added to homebrew-core: https://github.com/Homebrew/homebrew-core/pull/84805

> For some reason, I was under the impression age itself had some logic for keys.txt files.

An early version of the draft spec did include a default keys.txt path, and I implemented it in rage. However, during the beta phase discussions we made the decision Filippo described above, and I removed support for a default path in rage 0.5.0.

We've just finished our own updates to the draft, and are currently getting some feedback on it off-list before we publish the next version.

Answering what I can. Where something is an implementation detail, I'm referring to rage (which I'm obviously more familiar with).

1. rage tests arguments for validity as filepaths, and uses the file preferentially over treating the argument itself as a recipient format.

2. The header uses normal Base64. This was changed recently, and the examples likely need updating.

3. rage currently rejects unknown formats; I haven't implemented this part of the spec yet.

4. Based on the current contents of the age specification, it looks like limiting to standard Base64 characters is consistent.

5. Additional lines all need to be standard Base64 characters (i.e. consistent with the format of current recipient lines) if implementations are going to be able to skip unknown formats.

(Recipient lines are currently under-specified in the spec. I opened https://github.com/FiloSottile/age/issues/9 a while back for addressing this.)

6. The normalization notes are an artifact of an earlier ASCII armoring format. Now that the armor is (a strict subset of) PEM, there is no need for CRLF normalization, as the age format solely uses LF, and PEM (which can tolerate either) is only a wrapper around the age format and thus does not affect the header.

8. rage signals this via an I/O error in the library that will bubble up through std::io::copy; this amounts to truncation on a chunk boundary and a non-zero exit value.

Yes. age encrypts large files in 64 kiB chunks with 16 bytes of overhead per chunk, which strikes a balance between file size overhead and performance (particularly when seeking).

> ZCash is pretty explicit about the difference between shielded and transparent addresses....

This is an important point. Having two very distinct addresses (different lengths, different prefixes, different RPC APIs) makes it very obvious to users when they have the benefits of shielded transactions, and when they don't. Thus users make an explicit choice to forgo privacy when they use transparent addresses.

The problem of only 28% of transactions currently being shielded (https://explorer.zcha.in/statistics/network https://explorer.zcha.in/statistics/timeseries?hashrate=fals...) is a separate ecosystem problem, where usage of shielded addresses with third parties (like wallets and exchanges) requires them to use new APIs, instead of just interacting with the new block chain via the Bitcoin API. IIUC Monero has also encountered these issues, and it is something we are both working on improving.