Show HN: Radix sort big files in memory

Re:Show HN: Radix sort big files in memory by antender on Saturday May 09, 2020 @09:53PM (#23129722)

Was there any external requirement to specifically sort this file in-memory only? Why not just split the file into chunks (around 100MB), sort them as usual, and then k-way merge sort them after that. This, in theory, can be faster than allocating lots of RAM for radix tree, especially if you use SSD instead of HDD.

[ Parent | on HN ]

Re:Show HN: Radix sort big files in memory by antender on Sunday May 10, 2020 @12:01AM (#23130188)

After consulting with GNU sort manual: sort has -m option just for the case of merging presorted files, so you can test this by using 'split -l', then 'xargs sort' (to parallelize), then 'sort -m' to merge chunks

[ Parent | on HN ]

Re:Show HN: Radix sort big files in memory by afiodorov on Sunday May 10, 2020 @05:04AM (#23131278)

I agree that a significant proportion of time is spent on IO. Only 8m38s is actually spent sorting (out of 19m37s). However in the past my experiments have shown that using `sort -m` to sort chunks is much much slower than using `sort -S100%`.

[ Parent | on HN ]