News for hackers. Stuff that matters.
Re:Twitter plans to comply with Musk’s demands for data by srvmshr on Friday June 10, 2022 @09:28AM (#31693864)
The access comes with a cost and is pretty expensive unless covered by some academic grants. (Its about 10 grand USD). How expensive would Clickhouse be for data increasing about 1-2 TB per month? We are trying to optimize costs as the Twitter purchase already seems steep
Re:Twitter plans to comply with Musk’s demands for data by qoega on Friday June 10, 2022 @09:40AM (#31694031)
If you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns.

Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.

Re:Twitter plans to comply with Musk’s demands for data by srvmshr on Friday June 10, 2022 @10:38AM (#31694840)
Interesting. Would you mind talking offline (email in my bio). Would love to take some pointers