News for hackers. Stuff that matters.
Re:CrackStation's Password Cracking Dictionary by dfc on Monday February 18, 2013 @03:02PM (#5240287)
I am downloading the file now. I was a little surprised to see that you went with gzip. Given the target audience it is not that hard to imagine that the end user will have access to bzip2 or xz. With the size of the file why choose gzip? Hopefully `gzip -9` was used. I am curious about how much smaller xz/bzip2 will be. I will update this post once the file is downloaded.
Re:CrackStation's Password Cracking Dictionary by earthrise on Monday February 18, 2013 @03:14PM (#5240350)
I actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.
Re:CrackStation's Password Cracking Dictionary by dfc on Monday February 18, 2013 @03:22PM (#5240391)
Use xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.
Re:CrackStation's Password Cracking Dictionary by earthrise on Monday February 18, 2013 @03:29PM (#5240428)
Thanks, i'll run xz too!
Re:CrackStation's Password Cracking Dictionary by dfc on Monday February 18, 2013 @04:17PM (#5240718)
My download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make.

I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results:

    6830085 insane.msort
    1461684 insane.sort.xz
    1376548 insane.msort.xz
It is not a big difference but it is still a little better. For future releases you may want to do a case sensitive sort.

[1] Zebra comes before aardvark

Re:CrackStation's Password Cracking Dictionary by earthrise on Monday February 18, 2013 @05:46PM (#5241292)
bzip2 saved about 200mb:

gzip: 4293 bzip2: 4097

xz is still running

Re:CrackStation's Password Cracking Dictionary by agent86 on Tuesday February 19, 2013 @12:58AM (#5242880)
For anyone interested, p7zip set for -mx9 got the file down to 2.72GB on my Mac.
Re:CrackStation's Password Cracking Dictionary by dfc on Tuesday February 19, 2013 @09:25AM (#5244391)
I don't know about p7zip but the xz man page cautions against blindly using -9.