Re:Key/value is dead. Long live tuples: Pangool for Hadoop
by squarecog on Thursday March 08, 2012 @03:05AM
(#3678964)
You are doing regex matching in the Cascading code, but splitting on a character in the pangool code. The latter is obviously much faster. I don't know that that's the reason for the difference you observe, but it certainly can't hurt to fix that and make the user-supplied code more comparable.
[ Parent
| on HN ]
Re:Key/value is dead. Long live tuples: Pangool for Hadoop
by ferrerabertran on Thursday March 08, 2012 @11:28AM
(#3680299)
Indeed that regex was problematic because it had a bug itself. We replaced that line by RegexSplitter and updated the benchmark page. Please shout if you notice something else wrong. Thanks.
Re:Key/value is dead. Long live tuples: Pangool for Hadoop
by ivanprado on Thursday March 08, 2012 @12:09PM
(#3680541)
Just for clarify, split() java function is using regexp for the split as well. The code of String.split() is:
return Pattern.compile(regex).split(this, limit);
The benchmark seems fair to me.