News for hackers. Stuff that matters.
Re:Jetstream: Shrinking the AT Protocol Firehose by >99% by londons_explore on Tuesday September 24, 2024 @02:07PM (#41639312)
> Before this new surge in activity, the firehose would produce around 24 GB/day of traffic.

The firehose is all public data going into the network right?

Isn't that pretty tiny for a worldwide social network?

And the fact one country can cause a 10x surge in traffic also suggests its worldwide footprint must be tiny...

Re:Jetstream: Shrinking the AT Protocol Firehose by >99% by ericvolp12 on Tuesday September 24, 2024 @02:20PM (#41639451)
The 10x surge in traffic was us gaining 3.5M new users over the course of a week (growing the entire userbase by >33%) and all these users have been incredibly active on a daily basis.

Lots of these numbers are public and the impact of the surge can be seen here: https://bskycharts.edavis.dev/static/dynazoom.html?plugin_na...

Note the graphs in that link only show users that take a publicly visible action (i.e. post, like, follow, etc.) and won't show lurkers at all.

Re:Jetstream: Shrinking the AT Protocol Firehose by >99% by londons_explore on Tuesday September 24, 2024 @02:08PM (#41639322)
24GB/day is very much a "we could host this on one server with a bit of read caching" scale.

Every year or so, you plop in one more 10TB SSD.

Re:Jetstream: Shrinking the AT Protocol Firehose by >99% by ericvolp12 on Tuesday September 24, 2024 @02:23PM (#41639482)
Yes the actual record content on the network isn't huge at the moment but the firehose doesn't include blobs (images and videos) which take up significantly more space. Either way, yeah it's pretty lightweight. Total number of records on the network is around 2.5Bn in the ~1.5 years Bluesky has been around.

I aggregated some stats when we hit 10M users here - https://bsky.app/profile/did:plc:q6gjnaw2blty4crticxkmujt/po...

Re:Jetstream: Shrinking the AT Protocol Firehose by >99% by str4d on Tuesday September 24, 2024 @02:23PM (#41639475)
> The firehose is all public data going into the network right?

It's the "main subset" of the public data, being "events on the network": for the Bluesky app that's posts, reposts, replies, likes, etc. Most of those records are just metadata (e.g. a repost record references the post being reposted, rather than embedding it). Post / reply records include the post text (limited to 300 graphemes).

In particular, the firehose traffic does _not_ include images or videos; it only includes references to "blob"s. Clients separately fetch those blobs from the PDSs (or from a CDN caching the data).