News for hackers. Stuff that matters.
Re:Twitter plans to comply with Musk’s demands for data by imgabe on Thursday June 09, 2022 @09:41PM (#31689019)
> In truth, Musk probably only needs the “decahose” API Twitter makes available to some researchers, which is 10% of all tweets.

If you have the firehose, couldn't you just only sample every 10th tweet and make a "decahose"[1] of your own to work with?

[1] Shouldn't this be "decihose" since it's 1/10 not 10x?

Re:Twitter plans to comply with Musk’s demands for data by kareemsabri on Thursday June 09, 2022 @10:11PM (#31689207)
Yeah. A bit annoying to make a system to handle 5,000 events/sec (probably much more at peak) to throw away 90% of them though. Obviously you don't have an SLA on consumption time, but I imagine you can't lag too far behind assuming it's kafka or something.
Re:Twitter plans to comply with Musk’s demands for data by srvmshr on Friday June 10, 2022 @03:06AM (#31691006)
Genuine question: We signed up for the deci/decahose last month and for now making everyday dumps of about 50GB in flat files.

What kind of system architecture would be good to load & search through such dataset. We tried to explore Mongo Atlas but it is coming out very expensive. Other alternative is to throw away most metadata & just keep ID & Tweet.

Re:Twitter plans to comply with Musk’s demands for data by qoega on Friday June 10, 2022 @08:30AM (#31693122)
Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be.

By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves

Re:Twitter plans to comply with Musk’s demands for data by srvmshr on Friday June 10, 2022 @09:28AM (#31693864)
The access comes with a cost and is pretty expensive unless covered by some academic grants. (Its about 10 grand USD). How expensive would Clickhouse be for data increasing about 1-2 TB per month? We are trying to optimize costs as the Twitter purchase already seems steep
Re:Twitter plans to comply with Musk’s demands for data by qoega on Friday June 10, 2022 @09:40AM (#31694031)
If you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns.

Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.

Re:Twitter plans to comply with Musk’s demands for data by srvmshr on Friday June 10, 2022 @10:38AM (#31694840)
Interesting. Would you mind talking offline (email in my bio). Would love to take some pointers
Re:Twitter plans to comply with Musk’s demands for data by Nextgrid on Friday June 10, 2022 @03:17AM (#31691087)
If you don't mind a proprietary hosted service, try Google BigQuery.
Re:Twitter plans to comply with Musk’s demands for data by pram on Friday June 10, 2022 @04:27AM (#31691489)
Typical use case for Presto or BigQuery. Putting it in S3 and using Athena wouldn't take very long.
Re:Twitter plans to comply with Musk’s demands for data by nojito on Friday June 10, 2022 @05:58AM (#31692014)
Clickhouse will churn through it.
Re:Twitter plans to comply with Musk’s demands for data by webo on Thursday June 09, 2022 @10:26PM (#31689292)
>If the average length of a tweet (minus headers) is ~100 bytes of text, that’s only 50GB. You could fit it on a USB stick.

https://twitter.com/elonmusk/status/1534939289653592065

Re:Twitter plans to comply with Musk’s demands for data by ceejayoz on Thursday June 09, 2022 @10:31PM (#31689326)
You can't do bot detection with just the tweet text.

Tweets come with a lot more data than that.

Tweet object model: https://developer.twitter.com/en/docs/twitter-api/data-dicti...

Example payloads: https://developer.twitter.com/en/docs/twitter-api/data-dicti...

Re:Twitter plans to comply with Musk’s demands for data by webo on Thursday June 09, 2022 @10:41PM (#31689371)
Yeah, I was just highlighting how ignorant Musk is (sarcastically). This must’ve been what he meant by “hardcore software engineering.”
Re:Twitter plans to comply with Musk’s demands for data by Dylan16807 on Thursday June 09, 2022 @11:28PM (#31689647)
The example tweet, a somewhat large one with links and images, gzips down to 1.5KB by itself. Given that information, I think it would be reasonable to estimate 1KB per tweet in bulk.

500 million times 1KB is... still one USB stick, not even $50 https://amazon.com/SanDisk-512GB-Ultra-Flash-Drive/dp/B083ZL...

Or going by the retweeted quote tweet that's 40% bigger zipped, it's still well within 1TB which is a normal size for a flash drive.

I'm not very fond of Musk but this is a weak criticism.

Re:Twitter plans to comply with Musk’s demands for data by croes on Friday June 10, 2022 @12:39AM (#31690110)
Not it isn't. It's not about storage but about analysing the data. It's text so it could easily be compressed to smaller size but this doesn't help in knowing anything about the tweet author, if he is a bit or not.

That's like saying it's easy to check if a picture shows a bird because it's only 100kB.

Re:Twitter plans to comply with Musk’s demands for data by Dylan16807 on Friday June 10, 2022 @12:55AM (#31690210)
The tweet he replied to sure seemed to be about volume of data to me.
Re:Twitter plans to comply with Musk’s demands for data by jmeister on Thursday June 09, 2022 @10:59PM (#31689470)
Musk has one of the top AI researchers of his generation working for him(Karpathy). In general is rich enough to hire anyone he wants for consultation.

I’m boggled by the extent to which people mind-read Musk through his tweets. To me it’s obvious that his Twitter is an outlet for stress busting and banter, like smoke breaks.

Re:Twitter plans to comply with Musk’s demands for data by dubswithus on Thursday June 09, 2022 @11:10PM (#31689531)
Karpathy is on sabbatical and rumored to be interviewing elsewhere.
Re:Twitter plans to comply with Musk’s demands for data by naveen99 on Thursday June 09, 2022 @11:13PM (#31689558)
Hmm… with general AI around the corner, he should be getting paid more than steph curry.
Re:Twitter plans to comply with Musk’s demands for data by defterGoose on Friday June 10, 2022 @01:02AM (#31690244)
Re: your question, the answer is yes.