Re:Ask HN: Sorting massive text files?
by aw3c2 on Saturday August 14, 2010 @12:03PM
(#1603465)
I guess a line count has to run through the whole file, so obviously your storage is the bottleneck. Let's say your storage manages 100MB/s constantly for the whole 35GB it would take about 6 minutes.
Your 'cat' is not needed by the way, sort takes a filename as argument. And sort can '-u'!
It would be interesting if sort would fail on this (why?) or how long it would take.
Your 'cat' is not needed by the way, sort takes a filename as argument. And sort can '-u'!
It would be interesting if sort would fail on this (why?) or how long it would take.
edit: Why on earth are you doing this as root?