News for hackers. Stuff that matters.
Re:Twitter was down by sandworm101 on Thursday July 11, 2019 @03:42PM (#20415002)
>> he hit Control-C, and Twitter came back up.

Monolithic architecture. When I did security work I fought this every day. Moving away from it is a nightmare of technical debt and heated debate about who should control what. I'm reminded of a story from the early days of MSN. The legend goes that in the late 90s MSN ran out of one cabinet, a single server. The server had redundant power supplies, but only one physical plug.

Re:Twitter was down by hn_throwaway_99 on Thursday July 11, 2019 @03:45PM (#20415044)
> Monolithic architecture.

This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.

Re:Twitter was down by Dylan16807 on Thursday July 11, 2019 @06:50PM (#20416615)
It's not "nothing to do with". You're correct that a monolithic architecture does not imply that a single read query will lock the entire database. But it is a prerequisite.
Re:Twitter was down by hn_throwaway_99 on Thursday July 11, 2019 @07:21PM (#20416792)
Not really. I've seen more than one microservice-architected (admittedly, poorly) systems where, instead of the whole DB freezing up, just the one DB would freeze, but then all of the other microservices that talked to the frozen microservice didn't correctly handle the error responses, so now you had corruption strewn over multiple databases and services.

So, while true the failure mode would be different, "one bad query fucking up your entire system" is just as possible with microservices.

Re:Twitter was down by sho on Thursday July 11, 2019 @03:49PM (#20415098)
And of course this is standard practise. I've contracted on largish apps before (rails! Shock!) and of course we provided read-only replicas for BI query purposes. I wouldn't have provided production access even if asked.

Anything else is simple incompetence and the macro-organisation of the code and/or services is irrelevant.

Re:Twitter was down by sandworm101 on Thursday July 11, 2019 @03:56PM (#20415178)
If your website crashes because a single person ran a query, your system is too monolithic. You can have thousands of little microservices running all over the place, but a single query causing a fault proves that a vital system is running without redundancy or load sharing and that other systems cannot handle the situation. You have too many aspects of your service all tied together within a single system. It is too monolithic.
Re:Twitter was down by zbentley on Thursday July 11, 2019 @09:50PM (#20417454)
I think "monolithic" and "fragile" are orthogonal concepts.