Cutting out 99% of your auth backend traffic can dramatically change how you design it.
What is an auth server fundamentally?
It's a hash table lookup.
Which happens to be something that computers are designed to be really good at. I've never worked at a company where auth or storing user sessions was a bottleneck.
Auth might not be your primary bottleneck, but each page load you service has a latency budget and every millisecond spent being blocked on some auth call is a millisecond that could be spent doing else. And trust me, your database or crazy "solve it with redis" server isn't responding in 1ms, it's gonna be at least 20ms due to network overhead and the variance in latency on that request isn't always "20ms" it is "99% of the requests are 20ms and 1% are >>100ms". So now 1% of 10 billion requests (100 million requests) take way longer than 100ms... well that sucks...
Not to mention when your auth backend blows up and dies... your site is completely dead in the water. JWT can at least let you tread water for a while.
JWT is amazing. People that diss it don't understand it, or likely don't understand the tradeoffs they implicitly made with their own auth architecture.
I can tell you from my experience working at JP Morgan (the largest US bank) that authentication checks was the #1 root cause of major outages.
That is, except when it was #2. It was a close tie with network related issues depending on the season.
It's easy enough to understand that when sessions can't be checked, every application and service that need to check sessions are effectively 100% down.
Another hard learned fact is that (very) critical services or (very) high traffic services will simply never support remote sessions, because it's too slow and unreliable. (There are potential alternatives like JWT tokens or TLS authentication).
Computers are not good at network requests because physics says they can't be - it takes orders of magnitude more time for light/electricity to traverse even one side of a datacenter to another than it does to go from the CPU side of the motherboard to the RAM side.
For plenty of use cases, the additional network roundtrip is very affordable. But it's simple math that explains why this might easily become half or more than half of the total load on your systems if you want to "just do an additional (network-mediated) hash table lookup" on every request.
10ms minimum to go to the next rack. For 99% of the requests. For the unlucky 1% you'll get outlier performance due to.... well... you spend the next sprint figuring it out and the next quarter trying to solve that scaling bottleneck.... It could be network, it could be some weird design thing, it could be redis, who knows? But yeah... enjoy figuring it out instead of focusing on your product.
1% of all requests is actually a lot of requests. And if you think 1% of your auth requests aren't 10x the median... you aren't measuring it hard enough.
The signals going through the wire and back are 2000ns in a big datacenter. There are orders of magnitude more delays in the processing of those bits than their transmission through space via twisted pair or fiber.
It's not 10ms to go to the next rack though unless your network is broken. You're much closer to 100us (100,000ns).
This is a very common pattern in networking, people underestimate the speed of light and overestimate the speed of network equipment.
For example your auth server better be god damn awesome if it is gonna service every front-end page load. If even 1% of its traffic takes an order of magnitude longer than the median... that means 1% of your front-end traffic is gonna take an order of magnitude longer too. Lord help you if that fucker hiccups for a minute every so often... JWT shields most front-traffic from auth-server latency issues because most front-end traffic never needs to talk to the backend. (And no, your auth server isn't perfectly designed and yes 1% of its traffic will be an order of magnitude slower for whatever random reason)
Most web apps that use "normal" auth don't call an auth service. They drop an encrypted auth cookie that is checked on every page load and which is very fast. Session data and any centralised auth check (banned user for example) are simply database calls, which are no slower than the perhaps 10 or 20 other calls you are probably already making on your page.
> or aren't sophisticated enough to detect them yet
I'm sophisticated enough thanks!
JWT doesn't make sense if your app can simply make a direct database calls for authentication check. But if your app is composed of myriad of microservices which aren't allowed to query the database directly and must call another service to access data outside the caller's scope, then yeah the only option is making the authentication service fast enough to tank all the authentication checks or use jwt which makes any microservice able to confirm authentication status without querying the auntentication service.
Which is another reason why I think microservice architecture is a giant waste of time and productivity unless you're google-scale.
They're calls that you have to complete before you can start making any of the others (since it's not safe to start making database calls based on a user that hasn't been authenticated/checked for bannedness yet), so you're talking about more or less doubling or tripling your page's load time (check authentication, check authorization, then do the queries that give you the actual data) if you've got decently written queries.
That's not even arguing in good faith, or shows an extreme lack of experience. Auth/auth is routinely cached. Even if it's a database call, the query used there will likely be cached. Auth is a miniscule portion of a web page's performance.
Which completely undermines the argument for not using JWT. "We can't use JWT because then tokens would be valid for 5 minutes after the user logged out" - but if you're caching your auth for 5 minutes then you have exactly the same risks.
The real use of JWTs is when you want third parties to be able to sign their own API requests, that your API will accept using the very same machinery by which it accepts your first-party-client API requests. If you use JWT, you can just pass the third party a private JWT signing key to use, add the corresponding public key to your validation set for JWT decode on your backend, and everything "just works."
1. Log in with your service, receiving a JWT token
2. Send that token back to your service, as the auth key for authenticating as themselves with your service.
3. Also send that same token to the third party service, to authenticate requests against your service's account.
(One place where I just set this up the other day: allowing users of a crypto app to make authenticated requests to Infura via the MetaMask browser extension, where we're paying for the Infura credits to enable that. We send the user a JWT token; the client JS uses that token to build an Infura URL, and configures the MetaMask SDK with it. Infura accepts the requests as coming from "us" because it recognizes the JWT signing key we configured.)
Having the emitting entity sign their request in a standard format that can be checked by anything having with their public key helped a lot.
Sure there are some gotchas but it isn't that hard to read up on that once, configure your JWT check and then leave it alone.
Also, we pass additional claims in the JWT that avoids the need for the web service having to check certain permissions or auth status itself (like what resources the user has requested) and this is going to be a mess if you try and "just HMAC(SHA256())"
The point of JWT is that it's an ecosystem of stuff that all has built-in support for doing this. You don't have to arrange for this kind of signing "by hand"; on most services that have JWT "accept" support, there's a portal with a configuration field where you can just paste in a public key. Same as adding a line to ~/.ssh/authorized_keys, but for web requests.
(Yes, all of this is just a bad imitation of TLS client certificates. But browsers have a horrible, intrusive, non-syncing UX for client certificates. If they did with client certs what they do with cookies/passwords, nobody would have bothered to invent JWTs.)
Or use SHA3(secret || data), it was specifically designed to not have the pitfalls of sha2 which made HMAC constructions necessary along with being less computationally expensive. There's also KMAC built from keccak which has some advantages over simply hashing the concatenated key and message.
https://csrc.nist.gov/publications/detail/sp/800-185/final
https://crypto.stackexchange.com/questions/17735/is-hmac-nee...
Don’t “just” use some crypto construction you read on HN. Don’t invent your own thing. Find a well respected crypto library (eg libsodium) and look up your use case in the API documentation. Eg, for authentication libsodium exposes these API methods:
https://libsodium.gitbook.io/doc/secret-key_cryptography/sec...
Using sha3 as stated above is certainly something to consider for the numerous people out there implementing message authentication into their systems. It's simple and hard to mess up, if I was building a greenfields project today is probably the tool I'd reach for.
Libsodium is fairly complex and had it's own issues. Sha3 implementations in most languages are battle-hardened and simple to use.
[0] https://latacora.singles/2018/04/03/cryptographic-right-answ...
For example, what Sodium will actually do for the API your parent mentioned is HMAC-SHA512-256 ie it's using SHA512/256. But rather than give that as advice (and risk some developer either trying to re-implement SHA512/256 because their tool doesn't offer it, or worse, thinking SHA512 sounds strictly better so they'll do that) - Sodium just implements the thing you should do.
But if I were going to write a snarky "Right Answers" type post about this subject what it would say is that JWT almost certainly isn't the solution to a problem you have, it might be a solution to a problem you wish you had, but you likely have actual problems, now, and you should work on those.
Why do I say "wish" you had? Well as we see in a lot of these threads, immediately people talk about how it'll be a struggle to scale up the naive session approach when you have this big geographically distributed system. If your startup becomes Facebook or Google, you'll have to rewrite all this stuff. Why not instead use JWTs?
Your startup isn't going to be Facebook or Google. When you prove me wrong you'll have piles of cash for engineers to fix the scaling problems you knew about and, much more importantly, all the other problems you'd never dreamed of. You are attempting premature optimisation.
Percival, 2009: Use RSAES-OAEP with SHA256 and MGF1+SHA256 bzzrt pop ffssssssst exponent 65537. “””
Am I having a stroke?
dotnet auth tokens are also signed, they are stored in cookies, is that any different?
Wouldn't you just need a list of logged-out JWTs? And maintain entries until the JWT would have expired?.
And that's if you really cared. Some people would be fine with the client and server participating in the log out and a low-ish JWT expiry.
Sure, but that's not a real improvement. It's still stateful since you need to check the list with every request, you can't just verify the signature. Changing the logic from "must be in this list" versus "must not be in this list" does not give you much -- maybe some performance gain, depending on the situation.
> And that's if you really cared.
Well, that's my point. Developers may not care, but from the point of view of a security auditor, it's a vulnerability in your auth system, so the GRC/Security people do care, even if Foo start up doesn't. At some point, if Foo is successful, they will face this issue and need to support real logout to get the certifications they need to do business or be in compliance.
As someone who implements this exact solution, it actually gives you a lot. If your JWTs have a relatively small timeout window (say N minutes), then that means the number of expired tokens you have to keep track of is usually very small (e.g. for everyone who clicks the logout button with a valid token you have to keep that token around for at most N minutes), and on many websites the average user never makes an explicit logout request.
So that means that in many (most?) usage scenarios you can just replicate the list of recently expired tokens to your servers (easy with something like Redis) and then just check incoming requests against that in-memory list. Again, this isn't anything special or new, it is discussed in the article.
This is very different from storying the list of every valid JWT, which will often times be much bigger than you can store in memory.
You can verify the signature before calling the DB.
Similar security to "abandoned session" which is of course what a very large % of users do instead of "logging out".
There is risk of cloned session, but there's no law saying non-banking apps have to care.
This has the nice property that if you're forced to, adding a cache to store invalidated tokens with a ttl of 10 minutes is an incremental enhancement.
JWT comes in handy for most of your site where it isn’t such a big deal. That way you aren’t flooding your auth server with calls; all your web servers can just validate the token themselves up until a refresh is required.
Basically JWT let’s you have your cake and eat it too. Significantly less load on your auth backend while still letting sensitive parts of the site verify on every request.
Not having to block all your request waiting for the auth backend can improve your latency by a non-trivial amount. It also can make your auth server much less complex.
JWT is awesome. People just think it is "all or nothing" and implement it wrong.
That's generally my thought, too. In my career the authentication sophistication was always a trade off (effort vs security) based on how paranoid the org felt about certain things.
Having worked on PHP/Rails/.NET Webforms/Node.js apis/C# webapis the session management techniques are all very different.
Having run high traffic websites also, the ability to offload logged-in sessions into JWT was really, really nice. Not having to prune auto-generated sessions from disk of PHP applications is a nice thing to not worry about.
If one can steal a token from your browser's storage or can intercept your HTTPS connection, then sure, yes, you have a huge vulnerability even without JWT.
You notice odd behaviour so you log out to stop it but the attacker isn’t logged out.
Also with this setup log out of all devices is not possible! Again something you’d be keen to do if seemingly attacked.
In OpenID Connect, there are back channels that talk to each other so that all devices know you have logged out.
If you are using traditional auth then yes, you would need to track session state against the user so that when they access from a particular device, you could check whether they have already logged out.
Some more complex apps use a background thread to check auth status with the server.
It isn't unrealistic at all. A lot of attack vectors seem unrealistic to the victim until they happen. How realistic is it to think that a hostile actor can inject their code into a build process without touching the disk, then get that corrupted executable signed; then that signed executable is installed by 18000 users with root permission?
If you don’t want to maintain a centralized session store and you don’t want long lived tokens just set the expiry to 90 seconds. When you attempt a refresh, if the user is logged out it will fail.
I’m really confused by the either/or of “hit central session store on every request” or “tokens that live five minute too long are bad mmmkay”.
That sounds like a very complex system to cache session tokens. You may as well just cache normal session tokens for 90 seconds in your application servers. Same result, less complexity.
Although unlikely, I guess people want to know that clicking logout just does that and that there isn't a chance that some kind of hacker, colleague, nerfarious company or country couldn't continue to use your apps as you.
No need to have to persist state for each session, which is nice.
Sessions:Users are Many:1 typically so that's not as simple
You’re back to square one. Just use an opaque random session ID
You can use JWTs and have a "revoked_sessions" table that works like a normal "sessions" table. At the start of the request, check if the token is in the "revoked_sessions" table and deny access if it is. But there simply isn't a point; this has the same performance and latency considerations that a normal sessions table has ... but with many more possible ways to screw it up. "Sessions with extra steps."
I can understand the need to invalidate sessions server-side that might be malicious or fraudulent, to create a button in your app that says "Revoke all existing sessions", etc. And I can understand that other rules say a logged out session must not exist, and therefore you have to revoke the session. I especially agree if the JWT has sensitive roles in it, such as for an admin user.
But ... for most OAuth that uses JWT, I think keeping a short-ish expiry of say, 1 hour, combined with using cookies to store it is relatively okay. Perfect? No. But not terrible either, assuming you validate your JWT by specifying RS256 or whatever algorithm you expect, validate against cached JWKS and check both issuer and audience, issued date and/or expiry date, for example.
JWTs definitely aren't simple to validate unless you're lucky enough to use a really bullet-proof library out of the box. On the plus side, there really are only what, a half dozen steps to memorize about JWT validation. :D If new to JWTs, I highly recommend pasting one into https://jwt.ms and have a look. Yes, there's a lot to goof up, but they're relatively simple even so.
That said, I routinely use session cookies with a Redis KV session table to keep the JWT if I need it for other APIs etc. The one gotcha is making sure the random value for the session cookie doesn't already exist in Redis; using a timestamp-derived random string can help there, other than that, you're fine...
I would say though that this advice on JWT issuing might change in future: After all, Istio made running a PKI distrubution infrastructure a lot easier than before it came along allowing for mTLS between all the things, assuming your Kubernetes is secure, of course.
And the same advice on not issuing your own JWTs applies to session cookies like Flask's that are protected only by a server-side secret. Someday you'll need to rotate that secret and when you do, everyone's sessions will vanish forever. That's probably a good thing, and not a risk you'd see frequently, but when combined with the data exposure of keeping session data in potentially clear text (or base64 encoded text) which applies equally to JWT as to cookie stored sessions, well, it can feel a lot nicer to keep session data in KV storage. If you have really long JWTs, it's more practical too, compared to sending multiple cookies each under 4KB long to store a 7KB JWT with lots of extra fields and roles, etc.
Have the user change their password, and store datetime_password_last_changed on the user model. Then just treat any tokens issued before that time as expired. There are some use cases where storing tokens might be necessary, but for most startups this is probably not a good idea.
There's no extra DB request, assuming you need to lookup the user model on each request already.
Because you don't need to check the DB every request! That's the cool part! You only hit the DB for requests that matter!
For example, do you really need to check the DB to load this very page? If the auth token expiration is say, 5 minutes... does it really matter if a bad guy gets ahold of that token and loads this page? They can't post or vote because for those requests... you can pass the token to your auth server to be validated. 99% of your traffic is probably servicing the loading of this very page. 1% (or probably closer to 0.1%) is any kind of traffic that you actually care if the token hasn't expired within 5 minutes.
The problem people get into is they think security is black and white. It's gray. Some things need to trade latency for security... like write traffic. For 99% of the read traffic on HN? Who cares... even 10 minutes or more might be okay depending on the business.
People invent these complex pub-sub systems and stuff... they are working under the idea that all requests need to be super-secure. Most of it doesn't. I'd assert for most businesses.... 99% of the authenticated traffic could deal with a 5 minute max window after logout where somebody could maybe, theoretically impersonate somebody. And for the 1% of that traffic where it matters, you can always hit the DB.
"for most businesses" you're never going to have enough traffic that a classic session database (even stored in, gasp, MySQL) would be a bottleneck.
JWT definitely has it's places but it has so much complexity, so many tradeoffs and footguns that it should only be used when you have a specific problem with a standard session system that you need to solve, not as the default.
It's definitely more complex but at certain scales it might be worth it.
No expiry and no denylist means an attacker can cause problems until you patch your code.
I'm not disagreeing with you, it just seems to only cover a very specific type of attack, if someone else can mess with software. And of course if they can't it is unnecessary.
Cookies are managed by the browser, hence they are ultimately controlled by the user.
It really depends on the site and how much you care about having a window between logout and all the tokens becoming invalid. Most sites probably have large parts where it is a acceptable to have a few minute interval.
In the case of reddit, by not validating the auth token on every (non-sensitive) read request they could probably shave off 99% of the traffic going to their auth server. Better still it improves page latency as each request doesn't need to block on waiting for a response from the auth server. Faster page loads! And, since you were smart about it... all the stuff where it actually does matter that the token gets verified right away... you can do that too! It's win, win!
A lot of people mis-understand JWT or see it implemented incorrectly. JWT is pretty rad, honestly.
Yes.
> Like, what is the worst that is gonna happen in a five minute window between logout and the token expiring?
Complete compromise of your reddit account.
> But every read request for a normal user? You are telling me that those all need to re-verify every request?
For information that is not public, yes.
> JWT is pretty rad, honestly.
In the eyes of an attacker, absolutely.
Most security auditors/certifications/compliance requirements do not consider even a 5 minute lag acceptable for user driven logout. The definition of logout is that when the user presses the logout button, all the session data is invalidated server-side during roughly the time it takes for a request/response roundtrip.
Thus there is no benefit to using a non-opaque high entropy string for user sessions, even though there is a good benefit to using JWTs as API tokens for back-end services (esp. microservices) that do not have browser logout requirements.