> And the only way off without a mass password reset is a silent migration in the background: migrate the user when they login.. but we all know that will take months and you will never get 100% to login during the migration period.
Actually, not true. I can't speak for every auth provider, but FusionAuth and Auth0 both let you have the password hashes. If you know the algo (ask your provider!), you can load the hashes (and other ancillary password data like the salt) and your users will never be the wiser.
Here's a guide I wrote about how to migrate off of Auth0: https://fusionauth.io/docs/v1/tech/guides/auth0-migration/ The end goal of the guide is to move to FusionAuth, but the steps to get your password hashes out of Auth0 (the 'Exporting Users' section) will work no matter where you migrate to.
I maintained and extended a self rolled auth system built on top of django and it was a constant headache. We lost weeks of engineering productivity fighting with keeping the various libraries up to date and running into the usual "forgot password" edgecases.
I spent a few days with FusionAuth for a demo project and once I grokked it I really wished we had used it or something similar instead. Amazing abstraction layer to have at your disposal.
I want to chime in to this too.
If anyone from the Keycloak project is reading: keycloak is awesome, but the documentation is really the most painful part.
The main problem is that it assumes that people installing and/or configuring keycloak already are JBoss/J2EE experts.
Please assume knowledge of GNU/Linux systems, the OAuth/OIDC stuff, LDAP and SAML, but do not assume knowledge of JBoss.
Understatement of the year!
I have heard good and bad things about Keycloak, but I know one thing for sure: it's far better than rolling your own auth.
Good luck with your startup!
I believe that FusionAuth has a better developer experience. Everything is an API, the docs are regularly updated (though we can always do better). There's an easy way to set up developer environments to a known state (Kickstart) that I don't believe Keycloak has an analog for.
FusionAuth supports limited memory environments. We have folks running in 384MB of memory. From what I've read, Keycloak wants more resources, though Keycloak X is apparently a good alternative. I don't know if Keycloak X has feature parity.
Theming is easier with FusionAuth, more typical auth flows can be themed, and themes can be entirely API managed: https://www.keycloak.org/docs/latest/server_development/#_th... vs https://fusionauth.io/docs/v1/tech/apis/themes/
If you want to run thousands of tenants,, FusionAuth is better. Compare https://keycloak.discourse.group/t/maximum-limit-of-realms/8... vs https://fusionauth.io/blog/2021/03/29/seegno-thousands-tenan...
Both options offer support (Keycloak via the Redhat SSO package), but you can view our pricing without talking to anyone. I wasn't able to find pricing for Redhat SSO (that's usually not a good sign, but maybe someone who has engaged with them can add more, I could be mistaken and it could be super affordable).
FusionAuth has a 100% free as in beer edition with unlimited enterprise or social connections, users, and tenants. (There are certain usage restrictions; you can't package FusionAuth and resell it without a paid license, for example.)
Keycloak has a large community and is 100% open source. Those are definitely strengths that I want to acknowledge.
It's always good to evaluate software as critical as auth (or payments or notifications, for that matter) with a POC simply because everyone's situation is a bit different and it does get embedded in your systems (source: I picked Stripe for a startup and evaluated moving off multiple times to save money but didn't end up doing so in part because of effort and opportunity cost).
The only real downside of Keycloak IMO is the documentation. Because it covers so many bases, and is very extensible, the official doc is enormous. There are a lot of lightweight articles about how to use it, I'd use one of those to get started.
Some great things about it are:
- it's widely used, so you can google your way out of most problems
- it does everything. When someone asks if you have 2FA or support Yubikey - the answer is yes.
- it's very extensible. Great APIs, and you also have access to the underlying database should you really need it (we never have, I'm sure some do).
- it's open source. IMO this is a non-negotiable for an auth system
* IdentityServer (.NET)
* The ORY suite (hydra, etc)
* Gluu
I'd say Keycloak is the most common one I hear.There are about 15 FOSS packages listed here, but you'll need to weed through what is an auth server and what is not: https://en.wikipedia.org/wiki/List_of_single_sign-on_impleme...
My understanding is that you ended up writing java code to extend keycloak. Is that wrong?
Is keycloak resource hungry or have I heard wrong? Have you tried keycloak X? Is that better?
Is there a way to configure a dev environment? Is Terraform the preferred IaC solution?
How many realms do you use typically?
Depends what you mean by "extend". For many operations, there are REST APIs, e.g. to create a realm (somewhat equivalent to a tenant in the SaaS world - though see below), it's simply POST /auth/admin/realms. There are also more elaborate extensions such as login listeners that if I remember correctly, you would build in Java, but we don't need those.
> Is keycloak resource hungry or have I heard wrong? Have you tried keycloak X? Is that better?
We haven't found it to be disproportionally resource hungry at all. I'm sure there are scenarios we haven't exercised but resource usage not a problem for us. Not fond of the way vendors throw around allegations like that without hard facts.
> Is there a way to configure a dev environment? Is Terraform the preferred IaC solution?
Depends on your own preferences, we just bring it up in a docker container.
> How many realms do you use typically?
Everyone will skin this cat differently, but for us (SaaS):
- one big realm that most tenants use.
- one admin realm for trusted internal access (i.e. our support team).
- one big realm that candidates use (we are an HR platform)
- one realm per tenant that needs SSO. e.g. if they want to authorise login using their own GSuite account, they get their own realm with their own idp.
As someone else pointed out, having large numbers of realms is a weak point of Keycloak. I look forward to us having enough SSO clients that that becomes a problem for us :)
Always good to hear from someone who's actually used software rather than second or third hand perspectives.
You absolutely can migrate away from Auth providers. Auth0 (disclaimer: former employer) hashes passwords with Bcrypt, which is most likely the same exact algorithm you would choose. Bcrypt stores salts & rounds as part of the hash making those hashes fairly portable.
Even if you wanted to change algorithms, there are ways to do this so that it is transparent to the user, such as running the plaintext password through a new algorithm after a successful authentication and storing it in a new column.
Seeing as you have that it becomes a non-issue
You own the user table, but the auth logic is written by an expert.
If going with a third-party service, my advice would be to make sure you understand how to export the data out of the system, including hashed passwords using a non-proprietary algorithm.
I would expect that the way to do this sort of migration is to run it until the costs associated with maintaining the migration infrastructure exceed the projected losses due to churn. You don't need to worry much about auth provider pricing because these are typically billed on the basis of monthly-active-users, so even if you have a small number of users who aren't active, they'll hardly show up on your bill. When this pool of unmigrated users is small enough that you can tolerate churning a percentage of them, you can turn off your migration infrastructure and enforce a hard password reset.
That said, I don't know what kind of business has a significant percentage of users who churn because of a one-time password reset. I can't imagine any b2b contracts being canceled because the service provider enforced a password reset. Maybe retail sites are different?
With a password, I click the site I want to use, my password manager fills in the password for me, and I click "login" and bam, I'm done.
Even longer if you have greylisting turned on, it gets temporarily rejected which means you then have to wait N minutes before it gets accepted (assuming they retry sensibly and don't just spam retries every 2 minutes and get fail2ban'd.)
By which time it might well have timed out anyway...
...but the same is true if you would have a leak. So whatever I guess.
If you have a middleware that lets you swap providers, you can start off with a ready made solution and switch if necessary.
Obviously, this still means you need to maintain control over your data; but that's surely something a startup will do in any scenario, right?
This is some of the worst advice I've ever seen on HN. Don't take on risk you don't understand and cannot afford to mitigate. Don't be the next Equifax.
The reality is, though, it's never been easier than it is today to build secure auth - yes, it's still hard to do it right - but less hard than it ever has been before. So many security features are now a checkbox on a cloud provider's web interface that would have been a manual implementation of the underlying protocol a decade ago. So many languages have matured battle-tested OS libraries for handling the especially sensitive parts like crypto.
This notion that it is a "no-brainer" decision to happily trade your customer database and a % of revenue to remove the burden of stewardship seems crazy to me.
Hell, we use Laravel at work, which includes pretty robust auth features out of the box. It feels like such table stakes stuff for a web framework that I find the idea of paying for a third-party cloud service absolutely bananas.
It also includes pretty solid notification and payment solutions too, though these usually plug into services like SNS and Stripe and don’t really fit the “self hosted” bill.
Because otherwise their message falls flat on its face.
We use CSPRNG/SHA256/SHA512/AES/3DES/RSA primitives in .NET without any trouble. Microsoft made it pretty difficult to fuck things up as long as you don't sprinkle Math.Random in with your session token logic like a jackass.
I have a very vested interest in this (founder Clerk.dev) - but the exact opposite experience is what led us to start Clerk.
Auth was easy when it was just email verification and choosing the right hashing algorithm. I used to pull Devise off the shelf and get it going in 30 minutes.
But these days, the simple solution is far from complete. I grew most frustrated trying to setup the rest...
- Single sign-on / Oauth with proper de-duping so users can always sign in regardless of how they signed up
- 2fa so users can better secure their accounts
- Integrations with a leaked password corpus (haveibeenpwned) to prevent credential stuffing attacks
- Active device tracking and remote session revocation
You can launch without these things, but it's definitely impacting your overall security and the conversion rates through your sign up and sign in flows.
Proper password and session management have been made accessible for many years if you use any "modern" backend.
Something I once saw in production:
An otherwise normal usage of bcrypt, that "prehashed" the password with sha512 before passing the resulting hash to bcrypt. This is conceptually reasonable (although not very useful) for reasons I don't think are worth going into, but had two critical problems.
1. It technically broke bcrypt's side-channel protection against timing attacks (bcrypt runs in constant time, while sha512 does not).
2. The much worse issue was that the binary hash was being passed to bcrypt. If bcrypt encounters a null byte in the input, it considers that to be the end of the input and ignores everything after, c-string style. Every byte of the binary hash had a 1/256 chance of being a null byte, which means 1/256 passwords result in a hash with the first byte being null, which means bcrypt will accept any of them as equivalent to the others. You could login to 0.4% of accounts by guessing ~256 completely random passwords (and a larger number of passwords were similarly but less extremely weakened)
You can say "well that's their fault for mucking with the input to bcrypt" but I bet most people would believe that kind of operation was safe before it's explained to them why it's not. It's very easy to accidentally destroy the security of a crypto scheme without knowing it, and thinking that what you're doing is perfectly run of the mill and safe.
And then I'm reminded of the bug that was hidden in plain sight for 20 years in Programming Pearls[1] in their implementation of a binary search. This same algorithm was copied in Java and elsewhere, near verbatim. So, yeah, maybe don't roll your own searching either. The experts have enough difficulty.
Frankly, I don't trust most programmers to get anything right. People toss around "battle-tested" libraries and we all just take it for granted that that's what they are. But I recently found out one incredibly popular package for one incredibly popular language is doing something fairly dumb. Not insecure (that I can tell). Just simply unnecessary. They are signing session cookies. For reasons I could not fathom. Signing doesn't prevent cookie theft. It's just another layer that someone thought would add security over just sending a random string (hopefully your system/language's random generator is actually random). And that's how a lot of these mistakes happen. Let's add sha512 on top of our crypto. Because more = better, right?
Speaking of random, there was another popular package out there for generating UUIDs that has a github issue where someone discovered the package was generating collisions. Yikes.
[1] https://ai.googleblog.com/2006/06/extra-extra-read-all-about...
However, lest anyone get the wrong opinion, password hashing is a very simple, straightforward, solved problem and has been for years.
One person doing something monumentally stupid doesn't make Auth some kind of cryptic minefield.
I would wager that most people reading this, if earnestly asked by a coworker "do you think it's fine for me to hash a password before passing it to bcrypt? I want to be able to support passwords over 72 characters and bcrypt truncates its input." would answer something along the lines of "I don't see how it could hurt" rather than "that's dangerous because a binary hash would result in a large portion of the passwords being hashed as an empty string"
The engineers that originally implemented and reviewed this were not idiots, they just weren't security experts.
Try asking the following question on Twitter:
"While passing binary data, which example is safer for storing passwords?
[ ] $password | sha256 | bcrypt
[ ] $password | bcrypt"
What do you think the average programmer would say? Most would probably say they are either equal, or the first one, without knowing this specific thing, because most people don't implement their own password-hashing, they use library/framework provided ways that has been established as best practice already.
But, can't blame them really, the difference is marginal and innocent on the surface, but once you understand the implementation, you'll see the holes.
The yearly report of leaks in Fortune 500 companies should be proof enough of this.
EDIT: To elaborate. Crypto scheme is only one tiny facet of a successful authentication solution. Where do you store the hash? What language and stack are you using? What is the maturity of libraries available to you? What protocols? And many more seemingly tiny decisions. All it takes is a lazy developer that imports an insecure transient dependency or snoozes on a CVE.
What makes you think this foolish developer would not do the exact same thing before sending the data to an externally hosted auth service?
bcrypt accepts text input, not binary.
So if you use a pre-hash, you pass the hex or base64 output from the hash to bcrypt.
https://dropbox.tech/security/how-dropbox-securely-stores-yo...
> First, the plaintext password is transformed into a hash value using SHA512. This addresses two particular issues with bcrypt. Some implementations of bcrypt truncate the input to 72 bytes, which reduces the entropy of the passwords. Other implementations don’t truncate the input and are therefore vulnerable to DoS attacks because they allow the input of arbitrarily long passwords. By applying SHA, we can quickly convert really long passwords into a fixed length 512 bit value, solving both problems.
> Next, this SHA512 hash is hashed again using bcrypt with a cost of 10, and a unique, per-user salt. Unlike cryptographic hash functions like SHA, bcrypt is designed to be slow and hard to speed up via custom hardware and GPUs. A work factor of 10 translates into roughly 100ms for all these steps on our servers.
> Finally, the resulting bcrypt hash is encrypted with AES256 using a secret key (common to all hashes) that we refer to as a pepper. The pepper is a defense in depth measure. The pepper value is stored separately in a manner that makes it difficult to discover by an attacker (i.e. not in a database table). As a result, if only the password storage is compromised, the password hashes are encrypted and of no use to an attacker.
If you are considering the string copy that happens when the unhashed password is copied into the block to be variable time, probably you shouldn't count that against SHA-512 because there will be variable-length string operations before bcrypt starts its hashing too. Not to mention parsing it out of HTTP parameters or reading it from a GUI control. (If you really care about timing observations, perhaps you'd pre-hash the password on the client so what is transmitted is fixed-length).
Point 2 is of course a real problem. Though to be honest I've always regarded the NUL check and (sometimes, implementation-dependent) maximum length as unnecessary, arbitrary flaws in bcrypt - so bcrypt gets the blame. On grounds of purity and doing the right thing, I refuse to use raw bcrypt, and in fact I use: SHA-512 -> Base64 -> bcrypt when password hashing.
Does that make me wrong? Only if Dropbox is wrong (I hope they just forgot to mention Base64 in their old security article.)
I'm aware of this, and it's why there's a "technically" in my original point. Of course in practice it makes no difference, and I consider the better use of bcrypt's input space when prehashing to outweigh leaking a pretty useless amount of side-channel information for extraordinarily long passwords.
Though as a follow up anecdote, one of the engineers there had set a 300kB password to test the prehashing implementation, and just continued to use it (via password manager). He was potentially leaking a bit of timing information. I'm not even sure if a password that long made a timing difference that was detectable through network jitter, but if it did and some hypothetical attacker was able to learn that his password was hundreds of kilobytes long, well... good luck.
> in fact I use: SHA-512 -> Base64 -> bcrypt when password hashing. Does that make me wrong?
No, using a base64 representation of the sha hash makes your scheme perfectly reasonable. Also note that the base64 representation of sha512 is a bit longer than the typical bcrypt input length, so you're shrinking the collision space somewhat by truncating your sha hash. Certainly not enough that it would matter at all, just a point I find interesting. Hypothetically, a binary representation with a pass that removed or replaced NUL bytes would preserve more entropy, but we're well past the point where it matters and would be playing with fire to consider that complexity.
If you find collision space reduction interesting, then you might find it even more interesting that truncating SHA-512 makes it more secure not less.
This (and the higher rounds safety margin) is why I choose truncated SHA-512 over SHA-256 for this sort of application, even though the number of output bits is the same:
Check out the red boxes on https://en.wikipedia.org/wiki/Template:Comparison_of_SHA_fun...
Good luck to you!
Of course, if you only use third-party authentication for your service, it doesn't really matter what's in-between there, you can work around it.
However, now they've been bought by Okta, which has a pretty bad reputation.
So, I guess we'll get to find out eh?
I would say the same for Stripe, but at least they'll help you migrate off their platform. Auth providers cant help you because the passwords are hashed... You need the same algo or you cant authenticate using the data they have.
And the only way off without a mass password reset is a silent migration in the background: migrate the user when they login.. but we all know that will take months and you will never get 100% to login during the migration period.
Pick an auth provider and you better believe in their business as much as your own. You will incur damage when you leave.