Re:Don't MAWK AWK - the fastest and most elegant big data mu...
by fizx on Thursday September 10, 2009 @02:13PM
(#815822)
Depends. If you do a naive Ruby implementation, then you'll be CPU-bound quite quickly.
#!/usr/bin/env ruby
while line = STDIN.gets
puts line.split(/\s+/).first
end
This pegs my CPU at only 2MB/s, well below the IO capabilities of any modern system. I guess the tool you're using matters, which I think was the original point.
Re:Don't MAWK AWK - the fastest and most elegant big data mu...
by brendano on Thursday September 10, 2009 @07:09PM
(#816405)
I agree, 1GB is small. I do similar processing on more 100GB-ish tasks.
But I should say, this is a large enough dataset size that loading it all into memory is sometimes infeasible, at least in interactive interpreted environments like Python. That's an important boundary point.
That said, it's interesting that mawk is so fast.