News for hackers. Stuff that matters.
Re:Why Twitter didn’t go down: From a real Twitter SRE by wikfwikf on Tuesday November 22, 2022 @04:09AM (#33703681)
data architecture is bloat?

Have you implemented a system which stores hundres of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

Re:Why Twitter didn’t go down: From a real Twitter SRE by _3u10 on Tuesday November 22, 2022 @04:34AM (#33703832)
No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that.

Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…

Re:Why Twitter didn’t go down: From a real Twitter SRE by wikfwikf on Tuesday November 22, 2022 @05:06AM (#33704069)
There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
Re:Why Twitter didn’t go down: From a real Twitter SRE by pmoriarty on Tuesday November 22, 2022 @07:38AM (#33705092)
Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!).

Not tens of thousands at any one organization supporting postgres and clickhouse.

No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.

Re:Why Twitter didn’t go down: From a real Twitter SRE by rvnx on Tuesday November 22, 2022 @07:53AM (#33705213)
This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
Re:Why Twitter didn’t go down: From a real Twitter SRE by _3u10 on Tuesday November 22, 2022 @11:13AM (#33707346)
You’ve obviously never used FredDB
Re:Why Twitter didn’t go down: From a real Twitter SRE by bigiain on Tuesday November 22, 2022 @05:08AM (#33704075)
The Postgres engineers are not responsible for running their code on a shared platform supporting a billion users.
Re:Why Twitter didn’t go down: From a real Twitter SRE by iso1631 on Tuesday November 22, 2022 @05:39AM (#33704270)
Neither is twitter. Twitter isn't Facebook or Google. It's not even Reddit or Pinterest.
Re:Why Twitter didn’t go down: From a real Twitter SRE by badwolf on Tuesday November 22, 2022 @07:44AM (#33705142)
All those tweets with pictures and video that also get quoted and embedded in news articles that also need to be seen by people without an account.
Re:Why Twitter didn’t go down: From a real Twitter SRE by itsoktocry on Tuesday November 22, 2022 @06:02AM (#33704436)
>Where’s the 1000s of engineers for Postgres?

You're comparing Postgres to Twitter?

If some of the people making comments like this actually work in tech, then yeah, maybe there is a lot of bloat to be cut.

Re:Why Twitter didn’t go down: From a real Twitter SRE by 1500100900 on Tuesday November 22, 2022 @07:41AM (#33705120)
Postgres actually does have thousands of engineers but it's not easy to notice because they all share the same name.

https://www.citusdata.com/blog/2017/04/20/analyzing-postgres...

Re:Why Twitter didn’t go down: From a real Twitter SRE by lightbendover on Tuesday November 22, 2022 @09:42AM (#33706290)
I am entirely missing how Clickhouse solves the problem Twitter is solving out of the box, do you care to explain? Sub-question: how does Clickhouse provide monetization for a product?
Re:Why Twitter didn’t go down: From a real Twitter SRE by _3u10 on Tuesday November 22, 2022 @11:20AM (#33707435)
If Twitter was solving a problem it wouldn’t be in the situation it is.

It provides a house for your clicks, then you sell the clicks from your house.

In twitters case I would recommend they use checkhouse which will help them monetize their checks. Tumblr is already way ahead of them.

Re:Why Twitter didn’t go down: From a real Twitter SRE by donkeyd on Tuesday November 22, 2022 @09:46AM (#33706343)
It's bloat to devs who just want to write code. You're kinda confirming the point I was trying to make. Anyone on that list performs an important role at a company of their scale.

I'm currently the first and sole architect on a product that was built by only devs. I really know why I exist.