Dimsum is like head or tail, but pulls randomly from the entire file or stream

Re:Dimsum is like head or tail, but pulls randomly from the ... by teraflop on Monday November 26, 2012 @03:20PM (#4833788)

Doesn't look like it. I just tried running "yes | shuf -n 1" (using the latest version of GNU coreutils, 8.20) and its memory consumption increased steadily until I killed it.

It seems like this would be a really useful improvement, and I'm surprised that it doesn't already seem to have been requested on the coreutils issue tracker.

[ Parent | on HN ]

Re:Dimsum is like head or tail, but pulls randomly from the ... by malcook on Monday November 26, 2012 @06:18PM (#4834569)

did you try "yes | dimsum -n 1"?

In my hands, `top` shows resident memory increasing steadily too....

It is perhaps more instructive to compare output from, for example

seq 1 1000000 | valgrind --time-unit=B --pages-as-heap=yes --trace-children=yes --tool=massif --massif-out-file=massif.dimsum.100000.out.%p dimsum -n 1

with

seq 1 1000000 | valgrind --time-unit=B --pages-as-heap=yes --trace-children=yes --tool=massif --massif-out-file=massif.shuf.100000.out.%p shuf -n 1

in my hands, shuf is faster and uses less memory for this task.

How about you?

[ Parent | on HN ]

Re:Dimsum is like head or tail, but pulls randomly from the ... by snoble on Monday November 26, 2012 @08:29PM (#4835074)

sigh, memory leak. It's fixed in github. When camilo is around I'll get him to update the gem

[ Parent | on HN ]

Re:Dimsum is like head or tail, but pulls randomly from the ... by malcook on Tuesday November 27, 2012 @01:25PM (#4837972)

thanks - looking forward to the patch

[ Parent | on HN ]

Re:Dimsum is like head or tail, but pulls randomly from the ... by snoble on Tuesday November 27, 2012 @03:28PM (#4838778)

try a `gem update`. Memory performance should be much better now but I'm still curious about speed

[ Parent | on HN ]

Re:Dimsum is like head or tail, but pulls randomly from the ... by malcook on Tuesday November 27, 2012 @01:24PM (#4837960)

Agreed it would be.

So, let's pursue this: http://lists.gnu.org/archive/html/coreutils/2012-11/msg00079...

[ Parent | on HN ]