Apache Arrow and MinIO

Re:Apache Arrow and MinIO by idunno246 on Monday September 21, 2020 @01:13AM (#24540043)

I’ve had some bad luck with hbase for similar but not tried spark - it’s better optimized for writers than readers iirc. Pinterest[1] did an interesting thing where they worked directly on the hfiles. Everywhere I’ve worked I feel like I could pick any random data processing job that took more than a few minutes and get orders of magnitude improvement, there’s just so much inefficiency

[1]https://medium.com/pinterest-engineering/open-sourcing-terra...

[ Parent | on HN ]

Re:Apache Arrow and MinIO by anshumaniax on Monday September 21, 2020 @03:29AM (#24540749)

What do you mean inefficiency. We use the bulk upload feature and do billions of puts in an hour and our scans can go against 3 billion rows an hour. HBase scales linearly and we are already operating it on 5 times what we had designed it for

[ Parent | on HN ]

Re:Apache Arrow and MinIO by idunno246 on Monday September 21, 2020 @09:39AM (#24543250)

Ah I meant that as separate comment, not hbase specifically, but that data pipelines need updating over time. Generally since what’s worth optimizing for changes as size increase, and there’s always that years old pipeline that takes hours to run that with a few changes could be minutes

[ Parent | on HN ]