Re:Performance in Big Data Land: Every CPU cycle matters
by tobz on Thursday October 29, 2015 @09:23AM
(#10470750)
(Toby from Localytics here)
Yep, this is a great point. The data locality/reducing IO is huge, but the way things actually play out for us when data isn't segmented/partitioned properly, it chews up CPU/memory. This is a lot of why the post was geared around CPU usage: concurrency in Vertica can be a little tricky, and stabilizing compute across the cluster has paid more dividends than any storage or network subsystem tweaks we've made.
We're not at the PB/day mark, though, so there's definitely classes of problems we are blissfully ignorant on. :)
Yep, this is a great point. The data locality/reducing IO is huge, but the way things actually play out for us when data isn't segmented/partitioned properly, it chews up CPU/memory. This is a lot of why the post was geared around CPU usage: concurrency in Vertica can be a little tricky, and stabilizing compute across the cluster has paid more dividends than any storage or network subsystem tweaks we've made.
We're not at the PB/day mark, though, so there's definitely classes of problems we are blissfully ignorant on. :)