Command-line tools can be faster than your Hadoop cluster

Re:Command-line tools can be faster than your Hadoop cluster by sgt101 on Sunday January 18, 2015 @02:53PM (#8908599)

on a couple of GB this is true, actually if you have ssd's I'd expect any non compute bound task to be faster on a single machine up to ~10gb after which the disk parallelism should kick in and Hadoop should start to win.

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by KaiserPro on Sunday January 18, 2015 @03:17PM (#8908690)

Depends on the dick, depends on the storage.

HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things.

In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data.

In all these things, paying bottom dollar for hardware, forgoing support is a false economy. At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by radoslawc on Sunday January 18, 2015 @03:37PM (#8908774)

> Depends on the dick

not really, sorry I had to

back to the topic, HDFS is really somewhat waste of disk space, especially when used for something like munching logs

> At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.

depends, if you need monthly reports from logs, as long as you don't loose storage completely, then using even second hand hardware or decommissioned from prod is cheapest choice

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by KaiserPro on Sunday January 18, 2015 @04:35PM (#8909010)

Ahem

Disk....

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by simonster on Sunday January 18, 2015 @03:05PM (#8908646)

If you want disk parallelism, RAID 0 is probably easier than Hadoop.

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by cmurf on Sunday January 18, 2015 @03:16PM (#8908686)

That would depend on the data set and the strip size. Striping is good for streaming. Linear/concat of 2+ drives with XFS would be faster with a lot of files than end up in separate AG's on separate drives which can be accessed in parallel.

[ Parent | on HN ]