News for hackers. Stuff that matters.

decimusphostle

Example/Fabricated scenarios below

A simplistic view of the old scenario: Assume something like Google Analytics that tracks different websites traffic data. Previously a user could filter various website stats(unique users, time spent by users on a site, number of pageviews by user et al), by gender, by age, by device. So for example, interact with the data using a filter that picks all males(3), that are in an age bucket(assume 5 different age buckets) and that use a specific device category(assume 15 different device catgories). Here the cartesian product becomes 3(gender) X 5(age buckets) X 15 which would make it a maximum of 225 data-points per day, per website that is tracked.

And now the new scenario: Besides the number of dimensions going up i.e. more multipliers in the cartesian product above, the way a filter works is being tweaked as well. So previously where it was an AND across different dimensions(eg. all males AND 25-34 AND Windows Tablet users), now it is an AND between dimensions and an OR within a dimension(eg. all males AND [25-34 OR 35-44 OR 45-54] AND [Windows Tablet OR Windows Desktop] AND [Some fourth dimension option 1 OR Some fourth dimension option 2]... ). This makes it a combinatorial explosion i.e. its roughly 3! * 5! * 15! (even for the old scheme).

Ask HN: designing for interactive querying of large-scale high dimensional data

Hello Folks,I am looking to get some inputs on architectural design on data storage and processing for something new I am working on.I have large scale datasets (read a few hundred TB) which are used to generate data to power analytics. For the most part the analytics are usually data aggregations along different dimensions which are computed in an offline fashion (think Hive/Pig/Hadoop) and...

( Read More... | 2 comments | 3 points | on HN )