Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Reliance’s Ambani Lays Out Plan for Low-Cost Mobile Data in India. Parent comment
You are right about the volumes, but wrong about it being impractical.
The volumes with a relatively small opco:
- +-7m subs
- 250gb just for the protocol classification. *
- Then you also have url logs etc
Key factors that reduce the costs and investment:
- commodity hardware (with hadoop etc)
- distributed
- query patterns
- you do not need to store every single record. The data can be aggregated up to hourly, daily, monthly the older it is
This is what we did, data was aggregated which significantly reduced storage.
Tested various options: Hive, hbase, druid
Edit: * per day
There are no comments on this comment yet.
You are right about the volumes, but wrong about it being impractical.
The volumes with a relatively small opco:
- +-7m subs
- 250gb just for the protocol classification. *
- Then you also have url logs etc
Key factors that reduce the costs and investment:
- commodity hardware (with hadoop etc)
- distributed
- query patterns
- you do not need to store every single record. The data can be aggregated up to hourly, daily, monthly the older it is
This is what we did, data was aggregated which significantly reduced storage.
Tested various options: Hive, hbase, druid
Edit: * per day