What is a use case for minting tokens and one for not minting?

I've seen this referenced a few times before but I don't understand why minting tokens is bad.

Also, isn't creating a random key as a token the same as minting one? or what is the correct context here for "minting tokens"?

Thanks!

When I say "minting a token", I mean taking some data (e.g. your user name), adding some metadata (audience, expiry), and performing some operation on it (signing, encryption, authenticating) such that a different party will accept it as-is. (This is like "minting" because if you have a plausible-looking coin people would just accept it without having to go ask the issuer if it's a real coin via serial number or whatever.)

When you generate a random key, you have to go ask a trusted third party what the data associated with that key is (and, therefore, if it's still valid).

Minting tokens is bad because it's drastically more complicated, has tons of failure modes, and most of the time you end up doing tons more database transactions anyway that are way more complicated than set membership (i.e. is this still a valid token?).

I see. It makes sense.

In this context then, what are your thoughts on minting tokens using Macaroons [0]? I ask because they seem to be of much lower complexity than JWTs since they just hash the data so no encryption algorithm negotiation or anything of the like, however they still meet your definition of minting a token.

Obviously it would depend on a case-by-case basis to prefer macaroons over say a random token or vice versa, but in general is chaining HMACS still considered "drastically more complicated"?

My uninformed opinion is that verifying a hash shouldn't be that bad but my security expertise is pretty much non-existent so it would be great to be schooled on this :)

[0] - https://research.google.com/pubs/pub41892.html

If you're going to mint a token, macaroons are a great spec. But when designing a critical security system, the default should be the simplest possible thing, and DB lookup is still much simpler than macaroons. So, you have to have a big problem that macaroons solve first. In my experience, the cure is worse than the disease here.

Thanks a lot for the insight.

It's amazing how seemingly simple things can get so complicated when talking about security.

I did extensive research on various token types including Macaroons. While Macaroons look simple on the surface there are numerous edge cases that the verifier should take care of (Macaroons form a DAG because you can have multiple ones referencing each other). Plus there is symmetric decryption going on in case of third-party caveats. The caveat system can be very powerful (as caveats are just byte arrays you can use any system to encode claims). But there is no standard to encode them and that causes tight coupling between systems using Macaroons. I can go into more detail if that's interesting for someone.

I've read about Macaroons and the lack of a standard to encode caveats (mainly from the google group about macaroons) but I'm interested in your take, so more details would be great!

If you compare it to JWT in JWT claims are basically a string name (like "exp") and a JSON value. It's pretty simple. You need to check of the value for exp is a number and is greater than current millis. In Macaroons the entire caveat (a.k.a. claim) is just a byte array. The majority of libraries use predicates "X op Y" encoded in UTF-8 (e.g. time < xyz, account = 12345). That looks good, in JWT you don't have relation encoded, you just need to know that for exp it's "less than" and for aud it's equals.

Unfortunately that's where the simplicity of Macaroons end because even the official docs on libmacaroons have some weird choices, like encoding date in a ISO-like format (without timezone specifier at all). Also using UTF-8 is not that efficient if one could encode date as a number. Of course Macaroons are flexible and because the caveats are byte arrays you can use CBOR or whatever to conserve space. But then you lose compatibility. And you need compatibility if you want to utilize third party Macaroons (if a third party mints you a Macaroon with unknown caveat that makes the entire thing invalid). Third party Macaroons are the most powerful feature of the entire system but they introduce a lot of complexity: Macaroon references (look out for loops!), symmetric decryption and out of band communication needed to collect everything needed to authorize. That's why existing libraries allow only limited subset (e.g. no third party caveats referencing other third party Macaroons), and if you're writing a library (as I did) there is a lot of reverse engineering (e.g. libsodium is used in most libraries for symmetric encryption).

There are also some small things like de facto implementations using slightly different operations than the paper and a custom binary format. JWT just uses JSON and base64. (For the record I'm not a big fan of JWT).

What I like in Macaroons is the ability to further confine permissions. Sadly PAST doesn't seem to have something like this instead opting for a "safe JWT" way.

Awesome, thanks for the details.

One of the issues that I saw being discussed on the forums was precisely the lack of a standard format for caveats and even some suggestions to create one.

I think it's a very cool project but for some reason I don't think it has caught on. Do you think if someone came up with a suitable standard built it would see broader use? or what is your take on the lack of interest (IMHO) for this awesome idea?

Edit: Any way I could contact you to discuss this a bit more, if you are open to it? :)

Macaroons have many moving pieces, caveats being just one of them. Standardization would have to approach it from all edges. For example the binary format while simple is also "de facto" standard resembling protobuf but not exactly. Maybe CBOR would be a good idea (or a subset of it)?

As for caveats there are two forces at place: some want simple format ("X op Y") but there is another way that I've explored in a PoC - use simple stack-based script system (similar to Bitcoin Script [0]), then you can encode some really interesting properties inside your tokens (like requiring hashes of different properties, or signatures) so it would be kind of a meta-authorization scheme where you can delay the decision (e.g. require EC signatures for some sensitive operations). Of course this brings additional complexity but on the other hand using caveats in "X op Y" form doesn't bring any benefits over JWT claims.

[0]: https://en.bitcoin.it/wiki/Script

The ultimate reason why I abandoned Macaroons (after working for some prototypes and creating a JS library for them) is just the amount of complexity needed to work with them. And remember - code working with Macaroons is being executed before the request is authorized (that's what they are for) so this code need to be carefully audited and any bug can have severe consequences.

Compare that with JWT, you can write a verifier in simple code (JSON and base64 are built-in in any language) and simple is easier to audit. In Macaroons you first need to decode base64, parse custom binary format, check consistency (no cycles in third-party caveats), decrypt third-party keys. Moreover there can be multiple Macaroons for given ID, you need to check if at least one satisfies the request. Better - check all of them and then see if at least one works (to protect against side-channel attacks). So there is some inherent complexity in the entire stack. Removing it would require some substantial work. IMHO that's why they are not widely used. Oh, did I mention the existing libraries have some rather significant issues [1]?

[1]: https://github.com/nitram509/macaroons.js/blob/master/src/ma...

So to answer your question: I would gladly see some standarization effort, but it needs to be really thorough to have good effect. Unfortunately Macaroons are already plagued by old cruft (e.g. third-party caveats ID are called "cid" and first party caveats are also called "cid" but they are completely different) that no-one wants to touch not to break existing code.

If you don't mind we can keep the discussion here, it's good for others to see (I got into Macaroons because of one of these threads) and it's google-able :)

Just a small 'way forward' for those wanting to have the ability to revoke a JWT after I came up with a solution on my last project: A 'Gateway' - use OpenResty to verify the JWT ID stored in a redis cache using a proxy pass. When the Authentication service grants a JWT add its ID to this cache along with a way of identifying the user. That way the entire advantage/disadvantage of decentralised authentication is not fully weakened and OpenResty + Redis can be relatively fast.

Forgive my ignorance, but if you are going to all the trouble of storing the JWT id in a server side database for verification why don't you just store the JWTs' claims as well and just hand the client an opaque random id? Your gateway could do the lookup and supply the claims to your backend without the client knowing anything about it. You wouldn't even have to validate the claims, since they never leave your servers.

The best part of using JWTs is that you can validate without a central database. If you need a central database for sessions anyway you might as well store the claims in it.

Good solution! Openresty could create the JWT and add it to the forwarding headers, although the user would still need a cookie to maintain session with the proxy.

Only disadvantage I could see would be performance.

Congratulations, you’ve just invented session tokens. Don’t get me wrong: stateless tokens (like JWT) are terrible in part because they’re irrevocable, but then why bother reimplementing session tokens with JWT? Just use plain old session tokens instead.

They are not irrevocable - just harder to revoke.

If you want something that avoids storing all tokens, you can use a blacklist. You don't even have to check for revocation on any call - you could perfectly use short-lived (say 10 minute) access tokens and force frequent refresh using a refresh token and then only check the refresh token calls.

Whether you want to use it or not is a matter of making the right trade-offs. Stateful tokens are simpler to implement on the surface, but you have to keep in mind that the database lookup itself could be vulnerable to timing attacks.

Unfortunately, most database-based token implementations I've seen perform lookup based on the token string, instead of looking up the user and then checking all of the user's tokens, one by one.

And if you're not a small startup and actually have to handle loads north of 10,000 QPS (some of us do), these stateful tokens become quite expensive.

You blacklist your tokens in a cache and that's all.

So, for a secure system, the blacklist cache/service becomes a single point of failure (see also certificate revocation lists, ssl/tls).

I personally think renewable, short-lived tickets/tokens are a better evil - accept that a compromised session is valid for 10 minutes (5+worst-case/accepted clock drift).

A long-lived certificate can encode authorization + but needs a short-lived ticket to be valid ("an I'd card that says three star general and today's pass phrase").

> you probably still just want a random key in a database.

In general, I think that this is the wrong approach, because that means adding a database round-trip (which in a large system is almost certainly a network round-trip) for each and every API call. Notably, if the entire system is secured in depth (which large systems should be), it means adding a network round-trip for every layer of the API (e.g. one for the frontend server to validate the user, then another for a backend server to validate the frontend server). Using a stateful token[0] replaces that network round trip with a public-key or hash verification, which is much faster.

> revocation is still an issue

I think that generally revocation concerns are a bit overblown. Even with a database lookup on every request, there is a (small) amount of time that one is willing to act on out-of-date information (after all, the user's access could be disabled immediately after the lookup, before the action is performed). Almost every business has some window in which it is willing to act on old information: better to make it explicit than to leave it implicit.

I really like the long-lived stateless token-refresh token[1] and relatively short-lived stateful access token approach often seen in OAuth2 setups. E.g. an email provider might only bother refreshing a read-email access token every five minutes or so, but might wish to refresh a delete-email or send-email token more frequently (or require a database lookup each time).

[0] When I write 'stateful token,' I mean a token which is full of state; confusingly, some folks calls this a 'stateless token,' because the relying systems do not need to store or consult state.

[1] When I write 'stateless token,' I mean a token which carries no state and thus must be looked up in some form of database — a random key in a database would suffice.

> In general, I think that this is the wrong approach, because that means adding a database round-trip (which in a large system is almost certainly a network round-trip) for each and every API call.

In general, asking a database for set membership is not close to the slowest thing applications do.

> Notably, if the entire system is secured in depth (which large systems should be), it means adding a network round-trip for every layer of the API (e.g. one for the frontend server to validate the user, then another for a backend server to validate the frontend server). Using a stateful token[0] replaces that network round trip with a public-key or hash verification, which is much faster.

A handful of those set membership checks: still not the slowest thing applications do. (Also, I don't think it's a given that internal auth has to work the same way external auth does, for a bunch of reasons.)

> I think that generally revocation concerns are a bit overblown.

Without revocation, you can't log out of things, and you can't invalidate sessions after credential rotation. So unless you're saying tokens should be valid for some tiny number of seconds... in which case, it's questionable that you're buying a lot of performance for your drastically more complicated token scheme.

> Even with a database lookup on every request, there is a (small) amount of time that one is willing to act on out-of-date information (after all, the user's access could be disabled immediately after the lookup, before the action is performed). Almost every business has some window in which it is willing to act on old information: better to make it explicit than to leave it implicit.

Median response time is not comparable to usual token expiry times. If you do make them comparable, then you're making that DB check every time anyway. The entire point of the token is to do that less. You're right that it's implicit, but it's also "the fastest that it possibly ever could be", so that's not a bad kind of implicit.

If you really desperately want to make the check faster, why is JWT better than a local token cache?

> In general, asking a database for set membership is not close to the slowest thing applications do.

From Jeff Dean's list of numbers every programmer should know[0], a round trip within the same datacenter takes on the order of 500,000 ns; a main memory reference is on the order of 100 ns. How often will an application be making an order of magnitude more than 5,000 main memory references to service a request? Sometimes, sure.

In our experience online validation completely dominates our workloads.

> I don't think it's a given that internal auth has to work the same way external auth does, for a bunch of reasons.

Oh, you're completely correct. It has upsides & downsides.

> Without revocation, you can't log out of things, and you can't invalidate sessions after credential rotation.

I pointed out that I like the use of stateless tokens, which can be revoked. In many cases 'log out' can just be destruction of the token. And in many cases it doesn't make sense to invalidate access just because credentials have rotated (in many more cases it does, which of course is perfectly supportable with stateless tokens).

> So unless you're saying tokens should be valid for some tiny number of seconds.

I'm not: I'm saying that in many cases there's no business need to assure revocation within $SMALLNUM seconds, and that the business costs of online validation utterly dominate the costs of running a service.

> If you really desperately want to make the check faster, why is JWT better than a local token cache?

There are only two hard things in computer science … cache invalidation is one of them.

[1] https://gist.github.com/jboner/2841832

> From Jeff Dean's list of numbers every programmer should know[0], a round trip within the same datacenter takes on the order of 500,000 ns; a main memory reference is on the order of 100 ns. How often will an application be making an order of magnitude more than 5,000 main memory references to service a request? Sometimes, sure.

That argument is only valid if your application doesn't touch disk and doesn't touch the network to go talk to some other service anyway.

Do you have a concrete description of what "online validation" specifically means for you, and how long it takes? How long does validating the token take instead?

Just in case someone else reads this thread later, the point I was going to make:

- DC roundtrip: 0.5ms (per GP's own numbers)

- P256 ECDSA signature validation: 2ms [0]

[0]: https://www.cryptopp.com/benchmarks.html

> That argument is only valid if your application doesn't touch disk and doesn't touch the network to go talk to some other service anyway.

Why? Adding another source of “slowness”[1] isn’t free just because other sources of “slowness” already exist, especially if it’s already close to being unacceptably slow as it is.

I mean, sure, if the request takes ten minutes anyway and the validation check takes a few seconds, nobody will notice, but if I’m doing, say, a single non-local database lookup for the request then adding a second one for verification doubles the time it takes to service the request.

[1] I’m not saying that a network round trip and database lookup are slow, just that for the sake of this argument, they are being considered slow.

Do you have specific performance data for a comparable token validation?

I’m not really arguing against your point, just pointing out that your statement isn’t necessarily true. You said that if you do any network or disk access then validation will be negligible as it would be dominated by that. I’m simply saying that while this is probably true in most real world cases, it may not be so if for example both validation and normal request handling do a simple database query, then validation is 50% of the request time (or 25% if the request takes twice as long etc). Since the person you were relying to above said something about performance sensitivity, this overhead could be too much.

Having said that, I do believe that what you’re saying is the right approach for 99.9% of use cases and I would imagine that in almost all cases the performance hit really is negligible.

I gave specific examples of relative latency going the other direction in GP thread. But yes: not only is it not actually slow, it’s a much simpler engineering exercise to make it fast (see caching argument, same thread).

Is it literally definitionally impossible that minted tokens have a useful application? No, of course not. But absent very specific cases I’m going to argue for the thing that’s safe, fast & simple :-D

> There are only two hard things in computer science … cache invalidation is one of them.

I mean, it's a pithy saying, but TTL invalidation is a problem you have to solve with JWT too. Caching tokens is the easiest possible cache invalidation problem: definitionally, you know a priori exactly when the token is invalid.