Re:Twitter plans to comply with Musk’s demands for data
by qoega on Friday June 10, 2022 @08:30AM
(#31693122)
Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be.
By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves
Re:Twitter plans to comply with Musk’s demands for data
by srvmshr on Friday June 10, 2022 @09:28AM
(#31693864)
The access comes with a cost and is pretty expensive unless covered by some academic grants. (Its about 10 grand USD). How expensive would Clickhouse be for data increasing about 1-2 TB per month? We are trying to optimize costs as the Twitter purchase already seems steep
Re:Twitter plans to comply with Musk’s demands for data
by qoega on Friday June 10, 2022 @09:40AM
(#31694031)
If you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns.
Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.
Re:Twitter plans to comply with Musk’s demands for data
by srvmshr on Friday June 10, 2022 @10:38AM
(#31694840)
Interesting. Would you mind talking offline (email in my bio). Would love to take some pointers
Re:Twitter plans to comply with Musk’s demands for data
by Nextgrid on Friday June 10, 2022 @03:17AM
(#31691087)
If you don't mind a proprietary hosted service, try Google BigQuery.
What kind of system architecture would be good to load & search through such dataset. We tried to explore Mongo Atlas but it is coming out very expensive. Other alternative is to throw away most metadata & just keep ID & Tweet.