News for hackers. Stuff that matters.
Re:How Instagram scaled to 14 million users with only 3 engi... by Sparkyte on Saturday September 16, 2023 @03:34AM (#37532758)
I think this idea hurts the industry. Don't get me wrong. I think it is important to hire the right people, but if you don't hire enough people this 3 man job is 80-100 hours a week per for months to years.

Look how X has diminished in quality as Elon started slashing team sizes.

Then when you design more features, security and other various systems to serve the customers it will creep in complexity. You can not escape that no matter what you do.

Re:How Instagram scaled to 14 million users with only 3 engi... by frenchman99 on Saturday September 16, 2023 @03:41AM (#37532799)
I have noticed absolutely no difference with Twitter/X. I am a casual user, sure, but for me it seems to work well.
Re:How Instagram scaled to 14 million users with only 3 engi... by andrewinardeer on Saturday September 16, 2023 @04:08AM (#37532930)
There was a bit of downtime and some bugs did make it into their prod env after the restructure. These bugs were rectified fairly quickly too.

I generally agree with you on your comment. I am a casual user of X these days and the site seems to be humming along without any user facing engineering issues.

With his products such as Tesla, X and Starlink touching millions of people daily Elon is an easy to reach punching bag.

Even if he is somehow instrumental in solving the massive feat of putting humans safely on Mars there will always be people on the side lines having shots at him.

Re:How Instagram scaled to 14 million users with only 3 engi... by rpastuszak on Saturday September 16, 2023 @04:13AM (#37532959)
Anecdata for sure, but I use Twitter every day and features such as video upload just stopped working for me (failing silently), not to mention the site crashing or becoming completely unresponsive every week or so.
Re:How Instagram scaled to 14 million users with only 3 engi... by Marazan on Saturday September 16, 2023 @06:55AM (#37533686)
The entire multi-media infrastructure at Twitter is visibly creaking.
Re:How Instagram scaled to 14 million users with only 3 engi... by majani on Saturday September 16, 2023 @04:34AM (#37533038)
Funny enough, where I've noticed the most bugs is in the ad buying process. They want an active account (which many corporate accounts wouldn't qualify as), then they want you to be a Twitter Blue member(which requires a verified phone number), then my company's phone number wasn't accepted. I gave up after that
Re:How Instagram scaled to 14 million users with only 3 engi... by iLoveOncall on Saturday September 16, 2023 @04:37AM (#37533052)
I'm not even a casual user, I just go there when a friend sends me a funny tweet or an article point to one, and even I have noticed major bugs like comments not loading or other functionalities not working.
Re:How Instagram scaled to 14 million users with only 3 engi... by NavinF on Saturday September 16, 2023 @05:23AM (#37533282)
That's not a bug. If you're not logged in or your account looks like a bot, you can't see comments
Re:How Instagram scaled to 14 million users with only 3 engi... by anonzzzies on Saturday September 16, 2023 @07:08AM (#37533743)
> your account looks like a bot

That's not a bot if you are not a bot? I know it's 'normal', but I still it as a bug. AI classifying me as something I not is a bug.

Re:How Instagram scaled to 14 million users with only 3 engi... by NavinF on Saturday September 16, 2023 @08:08AM (#37534109)
Depends on the account. Eg if you look at unfiltered YouTube comments on youtube studio, you'll find many humans that behave like bots and post link to their videos in unrelated discussions
Re:How Instagram scaled to 14 million users with only 3 engi... by egman_ekki on Saturday September 16, 2023 @08:09AM (#37534116)
That’s exactly what a bot would say.
Re:How Instagram scaled to 14 million users with only 3 engi... by anonzzzies on Sunday September 17, 2023 @02:23AM (#37542077)
Doesn’t make blocking innocent accounts automatically less of a bug.
Re:How Instagram scaled to 14 million users with only 3 engi... by Marazan on Saturday September 16, 2023 @06:53AM (#37533680)
Media loading is dog slow these days. 10+ second waits for images in tweets to load.

That's the most directly obvious thing that has happened post takeover.

Re:How Instagram scaled to 14 million users with only 3 engi... by renegade-otter on Saturday September 16, 2023 @06:58AM (#37533696)
We don't know how that relates to the traffic. If the load caved, you need less capacity to handle it. The DeSantis launch campaign was a complete disaster, and only recovered mid-stream when 2/3 of the users left.

There is a case to be made that Twitter would have been profitable if it didn't torch money on unnecessary complexity, but the crashing ad revenue suggests Musk is not the business genius that you might expect.

Re:How Instagram scaled to 14 million users with only 3 engi... by weird-eye-issue on Saturday September 16, 2023 @03:42AM (#37532805)
> Look how X has diminished in quality as Elon started slashing team sizes.

what happened?

Re:How Instagram scaled to 14 million users with only 3 engi... by Sparkyte on Saturday September 16, 2023 @03:48AM (#37532839)
Back when he acquired twitter he fire 2/3rds the company.
Re:How Instagram scaled to 14 million users with only 3 engi... by falcor84 on Saturday September 16, 2023 @03:53AM (#37532863)
I think the interesting "what happened" is about the consequences - has there been some objective drop in uptime/performance/errors or some other technical metric?
Re:How Instagram scaled to 14 million users with only 3 engi... by Sparkyte on Saturday September 16, 2023 @03:59AM (#37532897)
Yes actually.

And one massive security breach.

Re:How Instagram scaled to 14 million users with only 3 engi... by andrewinardeer on Saturday September 16, 2023 @04:10AM (#37532945)
Even before he acquired the company they had security issues. Someone even managed to tweet from his account with a BTC scam.
Re:How Instagram scaled to 14 million users with only 3 engi... by leereeves on Saturday September 16, 2023 @04:09AM (#37532937)
Could you be more specific?
Re:How Instagram scaled to 14 million users with only 3 engi... by Sparkyte on Saturday September 16, 2023 @03:56AM (#37532881)
@falcor84

We hit maximum reply lengths but yes there has been. And a massive security breach.

Re:How Instagram scaled to 14 million users with only 3 engi... by wfme on Saturday September 16, 2023 @04:02AM (#37532908)
Which security breach are you referring to? I'm not aware of any major ones that have happened since the acquisition, but this one seems to be the one people point to even though it happened prior. https://privacy.twitter.com/en/blog/2023/update-about-an-all...
Re:How Instagram scaled to 14 million users with only 3 engi... by drno123 on Saturday September 16, 2023 @03:56AM (#37532875)
No, what is the consequence, how has quality of Twitter diminished? As a “normal” user I can not observe any degradation of Twitter service.
Re:How Instagram scaled to 14 million users with only 3 engi... by Sparkyte on Saturday September 16, 2023 @04:02AM (#37532906)
Sure, but I feel like this was because of feature creep more than anything else.

Most common thing people use it for is posting and reading. If you fake feeds as if they are real-time then the viewer will never know there was a system outage.

I am almost positive there was a massive security breach too.

Re:How Instagram scaled to 14 million users with only 3 engi... by blackoil on Saturday September 16, 2023 @04:41AM (#37533074)
> I am almost positive there was a massive security breach too.

That's a very serious allegation, as it would be illegal to not report it. Can you add some details?

Re:How Instagram scaled to 14 million users with only 3 engi... by thebears5454 on Saturday September 16, 2023 @04:44AM (#37533088)
Very famously he had Ron DeSantos on to announce his presidential campaign, but his website didn't work at all.
Re:How Instagram scaled to 14 million users with only 3 engi... by 3np on Saturday September 16, 2023 @09:47AM (#37534724)
Experiences vary wildly between and IP addresses and browser profiles. The variance increased drastically post-takeover.
Re:How Instagram scaled to 14 million users with only 3 engi... by weird-eye-issue on Saturday September 16, 2023 @01:08PM (#37536584)
This response is about what I expected
Re:How Instagram scaled to 14 million users with only 3 engi... by thilog on Saturday September 16, 2023 @03:43AM (#37532810)
Twitter most likely hasn't been able to retain the "right" people.
Re:How Instagram scaled to 14 million users with only 3 engi... by yieldcrv on Saturday September 16, 2023 @04:05AM (#37532920)
Only people that don't want to get deported during a tech hiring slowdown
Re:How Instagram scaled to 14 million users with only 3 engi... by Sparkyte on Saturday September 16, 2023 @03:47AM (#37532836)
This is true.
Re:How Instagram scaled to 14 million users with only 3 engi... by Aeolun on Saturday September 16, 2023 @03:44AM (#37532812)
Having only 3 people forces you to be very judicious about what you do. For most companies that’s a good thing.
Re:How Instagram scaled to 14 million users with only 3 engi... by EspressoGPT on Saturday September 16, 2023 @03:44AM (#37532818)
> Look how X has diminished in quality as Elon started slashing team sizes.

He had a point there though: He said that there seem to be "3 managers 'managing' one engineer", and I believe this is a common problem in the industry. VC-funded startups are terribly overstaffed and over-inflated.

Re:How Instagram scaled to 14 million users with only 3 engi... by rhuru on Saturday September 16, 2023 @04:30AM (#37533011)
Musk's actions have certainly damaged Twitter and its engineering. When you set the house on fire of course some termites will get killed but then you might also kill some babies and the family dog.

A lot of tech companies have bloat in the form of AI ethics people, DEI people and so on. They need to go. But Musk probably hurt twitter a lot in short term by firing a lot of engineers and making it a place that made people unhappy.

Re:How Instagram scaled to 14 million users with only 3 engi... by csomar on Saturday September 16, 2023 @04:33AM (#37533031)
> AI ethics people

These do actually have a proper job. They do ethics laundering for the tech companies and are very valuable.

Re:How Instagram scaled to 14 million users with only 3 engi... by mk89 on Saturday September 16, 2023 @05:17AM (#37533250)
He is squeezing everything as much as he can.

Introduced failing ideas that made him ridiculous world wide, ... to finally hire a CEO.

And now he is using the platform to influence elections and events - free for all. Sure.

Re:How Instagram scaled to 14 million users with only 3 engi... by dingdong33 on Saturday September 16, 2023 @07:04AM (#37533722)
Our company has a position like a DEI director, and honestly, it baffles me. All they seem to do is push what some might call 'leftist woke propaganda' in their never-ending meetings, and I question whether our company's resources should be invested in that.
Re:How Instagram scaled to 14 million users with only 3 engi... by agentgumshoe on Saturday September 16, 2023 @07:27AM (#37533848)
It's just 'brand.' Companies don't generally give two stuffs about their staff, as you find out when anything like unions or HR complaints come up.
Re:How Instagram scaled to 14 million users with only 3 engi... by esperent on Saturday September 16, 2023 @07:18AM (#37533807)
Could you give some concrete examples from this DEI director of the kind of things you're calling 'leftist woke propaganda'?
Re:How Instagram scaled to 14 million users with only 3 engi... by rhuru on Thursday September 21, 2023 @05:12AM (#37595139)
I am not OP but one of the things I have seen these people do is to insist on removing words like "nitty gritty" from our developer documentation by calling it racist.
Re:How Instagram scaled to 14 million users with only 3 engi... by bornfreddy on Saturday September 16, 2023 @07:09AM (#37533751)
TIL: DEI == Diversity, Equity, & Inclusion (if anyone else didn't know).
Re:How Instagram scaled to 14 million users with only 3 engi... by rhuru on Thursday September 21, 2023 @05:11AM (#37595124)
Ignorance is bliss. If you do not know what DEI is it is a bit like not knowing how to be an overweight. You are better off not knowing it.
Re:How Instagram scaled to 14 million users with only 3 engi... by thebears5454 on Saturday September 16, 2023 @04:43AM (#37533079)
There is not a single VC that is three managers to one engineer
Re:How Instagram scaled to 14 million users with only 3 engi... by dijit on Saturday September 16, 2023 @05:37AM (#37533344)
Team Lead, Scrum Master and Director.

Not to mention HR managers.

I have seen situations where there are 10% engineers to 50% "assorted management" in tech companies. (the remaining 40% being a mix of sales and support staff such as office management).

Re:How Instagram scaled to 14 million users with only 3 engi... by GaelFG on Saturday September 16, 2023 @05:45AM (#37533386)
"I won't give names but trust me", VC funded tech companies with 3-10 bros 'who have or not a vague technical background but more see themselves as high level thinkers' between founders and their first hires for 'key strategic roles' followed by an engineer and one to three interns to fill production role are a common reality, at least on startup scenes I attended. In the startup process 'marketing' is also glorified (not that i disagree it's importance) so once you have one or two guy who can make demos, hiring marketing people is often the next priority.
Re:How Instagram scaled to 14 million users with only 3 engi... by mk89 on Saturday September 16, 2023 @05:31AM (#37533313)
When you own a company and fire someone to cut costs nobody is gonna say anything, of course. It's the sad reality.

When you cut the company to 1/3 and keep foreigners because of their visa status, nobody is gonna say anything of course, but that says a lot about you!

I do not believe that half the company was just "overstaff". I have been in situations where 1 manager had 1 reporter, but they were single cases - it can't be spread to the entire company and nobody does anything.

Re:How Instagram scaled to 14 million users with only 3 engi... by bornfreddy on Saturday September 16, 2023 @07:03AM (#37533721)
> When you cut the company to 1/3 and keep foreigners because of their visa status, ... that says a lot about you!

That... you care about people regardless if they are foreigners and that you try to help those that would have the most problems if they were let go, especially as these problems are a consequence of your hiring of them?

I'm not familiar with the story, but from the way you presented it it sounds like a proper thing to do.

EDIT: not taking Musk's side, just pointing out the issue with the parent's argument.

Re:How Instagram scaled to 14 million users with only 3 engi... by kyriee on Saturday September 16, 2023 @08:08AM (#37534108)
Or… heart me out, maybe it’s because Elon knows they have the most to loose and are thus more likely to do whatever he wants to not get kicked out of the country. Elon made some statements that made his disdain for the “laptop class” quite clear. He is no saint, or genius. He’s mostly just an extremely ruthless smart man.
Re:How Instagram scaled to 14 million users with only 3 engi... by bornfreddy on Saturday September 16, 2023 @10:26AM (#37535045)
Sure, that is also a possible (even likely) explanation. As I said, I don't defend Musk - my opinion of him is quite low and is getting lower and lower. Just wanted to point out the trouble with the argument. Anyway.
Re:How Instagram scaled to 14 million users with only 3 engi... by mk89 on Saturday September 16, 2023 @02:56PM (#37537553)
He has finally showed the world his true self. I am happy about that - I don't want another capitalist sanctified for being a genius, which he is not. The more this becomes clearer the better for everyone.
Re:How Instagram scaled to 14 million users with only 3 engi... by mk89 on Saturday September 16, 2023 @02:52PM (#37537515)
Sorry, you're right. I was try to point to different direction: you keep foreigners because you can exploit their visa status. :)
Re:How Instagram scaled to 14 million users with only 3 engi... by gumballindie on Saturday September 16, 2023 @04:43AM (#37533080)
> 3 man job is 80-100 hours a week per for months to years.

Is it though? Small, lean, teams have fewer processes, less distractions, better communication, and more flexibility in what they can do. I've been in such teams and built such scalable systems and there was nothing 80-100 hours about it. It turned worse once the company was acquired and management and "specilised" workers were brought in.

Re:How Instagram scaled to 14 million users with only 3 engi... by pcchristie on Saturday September 16, 2023 @05:01AM (#37533182)
> Look how X has diminished in quality as Elon started slashing team sizes.

Has it? I've had the exact same experience.

Re:How Instagram scaled to 14 million users with only 3 engi... by coldtea on Saturday September 16, 2023 @05:09AM (#37533216)
>Look how X has diminished in quality

Ideologically or from a service perspective? I don't see any noticable drop in the latter. Some disruption is expected as huge parts of the team is fired/leaves, but I see it going on in business as usual otherwise.

Re:How Instagram scaled to 14 million users with only 3 engi... by kristopolous on Saturday September 16, 2023 @05:19AM (#37533254)
> You can not escape that no matter what you do.

The point is you can.

Choosing not to do is probably more important than being able to do.

Organizations are reflected in the products they create. We shape our teams and thereafter they shape us.

This doesn't mean brain teasers or other arbitrary metrics with standard bell curve distribution so you can pick the statistical outliers and claim you've done this. That's totally completely horrendously wrong because that's not what you're fitting.

Those are filters that produce stochastic results with merely the statistical properties of these rules of thumb.

If you're looking for a programmer, here's a better test: think if some famous programmer walked in and sat down to do your process. Could they pass? If the answer is "dice roll", meaning you'd say, turn down Rob Pike or Larry Wall, then you're doing it wrong

Re:How Instagram scaled to 14 million users with only 3 engi... by ben_w on Saturday September 16, 2023 @05:41AM (#37533366)
> Look how X has diminished in quality as Elon started slashing team sizes.

True but different: even ignoring how many of them were customer support and moderation, once the tech stack gets complex, you can't just snap your fingers and act like it's a simpler stack, not even if a fresh 3-good-graduates team can reach feature parity with your now-1000-person team, when they've had 6 months from `git init` and you've been at it since they were in Kindergarten.

Re:How Instagram scaled to 14 million users with only 3 engi... by teakie on Saturday September 16, 2023 @06:15AM (#37533520)
this is what work is.

efficiency shrinks by more team also you get other people. if you are working on something you stay alive until it's done.

≠ a job

Re:How Instagram scaled to 14 million users with only 3 engi... by Mizoguchi on Saturday September 16, 2023 @06:59AM (#37533700)
Considering the hostile takeover of Twitter, the layoffs, lack of leadership and Elon pulling cables out of servers, the platform has been working exceptionally well, at least in my experience. Either it was built like a tank or some of the people working there are extremely competent. To be fair is not like Twitter has never been prone to issues. I'm an old enough user to remember the fail whale and many of the great outages of Twitter's first decade. It didn't became stable until maybe 5 years ago and even recently just prior to Elon coming in they had another hours-long downtime.