The xkcd comic assumes a dictionary-based attack. It points out that choosing 4 words at random from the 2048 most common words provides more entropy than choosing one word at random from the most common 65536 words and performing some letter substitutions on it. In both cases, it is assumed the attacker knows the formula used to create the password.
If your formula for creating a password is "Choose a printable ASCII character at random, then repeat", you only need a password that is 7 characters long to beat the "4 random common words" formula (95^7 > 2048^4). But the percentage of people who construct passwords by choosing random printable characters rounds to zero.
Leveraging song lyrics can be a helpful step in this direction: simply feed in the first letter of each word to create your password. Of course, you would need a fairly long lyric/phrase to exceed the benchmark noted above, but on a scale of 'hunter2' to '8%jFb#P", I'd say it's not bad :)
The place I'm working for right now used to have no password policy for the end users. We also had a feature that allowed them to link their Twitter accounts so that they could automatically share content out to their networks.
Eventually what had happened was that people blamed us for their Twitter accounts getting "hacked". What actually happened was that they set a dumb password (like "password") on their account with our service, authorized us to post to Twitter on their behalf, and now when someone hacked their account with us they could now post to the victim's Twitter account.
We started taking plenty of heat for this, so eventually we decided to impose a password policy (a sane one, at that) and this problem eventually went away. The users of our service aren't particularly tech savvy, so blaming them for their shoddy practices wouldn't have done us any good. It might have made sense to try to fight this if we were a company like Okta, that sells security as a service, but we don't so we had to make the decision to enforce password requirements to just stop the all the bad reviews in the app store and social media hate we were getting.
If I roll a dice, and ask you to guess the result, it will take you 3 guesses on average to get the result. If I ask you to guess the random word I've chosen, it will take you around 1000 guesses (if I have a small vocabulary).
Entropy is calculated as ln₂(x), where x is the number of possibilities. With four common words from a list of 2000:
4 × ln₂(2000) ~= 44
as used in the comic.If, instead, the password was taken to be 28 random characters from [a-z ], the entropy would be
28 × ln₂(27) ~= 133
This is the overestimate that the artist didn't make.With a genuinely random alphanumeric password, like 0LsW-nT5#^kQ, the entropy is higher:
12 × ln₂(92 keys on the keyboard) ~= 78.
but the point of the XKCD was to show a memorable password.
I'd much sooner read a piece discussing the reasons developers so often make this design choice. Is it a thoughtless formula, a common misconception about how brute-force attacks are conducted, or something else?
I'm far from an expert on the subject, but as an aside, I'd like to point out that the XKCD comic on the topic seems inconsistent with what I know about how brute-force attacks are usually conducted. To my knowledge, these are usually dictionary-based attacks, which would significantly decrease the "search space" to discover even a long, lower-case sentence. Thus a password with random characters and shorter length might be more secure in practice than a longer password composed of English words.
Edit:
I've been informed that my layman's speculation on the comic was mostly wrong. As I said below, "sorry to be on the other side of the 'infuriating argument with someone who doesn't know information theory or security.'"