News for hackers. Stuff that matters.
Re:Apache Arrow and MinIO by anshumaniax on Monday September 21, 2020 @03:26AM (#24540735)
Really we went the exact other way for one of our use cases which required table scans and spark just lost. HBase just stores bytes in sorted order and once you know how to optimize for storage lots of wins can be achieved. I guess the use case here was aggregation so can definitely see some spark advantages.
Re:Apache Arrow and MinIO by nl on Monday September 21, 2020 @06:39AM (#24541814)
Yes, HBase index scans are very fast.

(Any index scan is extremely fast really. You can build your own indexes as separate Parquet files if you want to avoid HBase for some reason)