For example, speaking completely independently from the problem here but putting on my performance hat, I would expect that if we take all things as a given (“no major architectural changes allowed”) Teams should not take 10 seconds to start up to show UI. We know that the tech stack they’re using isn’t fundamentally that bad. This suggests that there is probably a handful of things they are doing that are low-hanging fruit but haven’t been identified yet. Once they find those and bring down the time to something more reasonable like a second things will get much harder for them to fix.
So I highly doubt the people driving this initiative are just "unfamiliar with how to improve their software's performance". Something else is going on here: maybe they were saddled with something so bad it could only be fixed so much, maybe there's some technical hurdle we're just not appreciating, maybe it's an organizational problem (this is my guess). But Slack has a similar problem, so I doubt it's the first possibility (the 1.0 of the app just happened to be botched). There's something going on here
Here's my wild uninformed guess: Teams has lots of features/integrations, and they're constantly expanding because it's in the limelight right now, so they've got too many engineers on it working on different features at the same time and not talking to each other enough. The system is now too sprawling to be considered (or optimized) as a whole, which means there's tons of redundancy and toes-stepping going on. Shipping the org chart, etc.
(Ask me where this vivid picture came from)
They started with Electron, and are now moving to WebView2. When the project started Edge was still not Chromium based (it was the IE fork) so they couldn't even use that.
"We always run at 60 fps, because if we finish the game, then try to optimize our way to 60, we'll never hit it."
>> This suggests that there is probably a handful of things they are doing that are low-hanging fruit ...
The Quake annecdote suggests the opposite. If I had to guess (from personal experience with web development), it's probably a million-billion little things that, when added up, make your whole project a slow, gelatinous pile.
That's not to say there's not also low-hanging fruit that might make switching between conversations take less than a second, but .. it should take .. like .. a short amount of time, like 16.6ms, including the network trip.
(note that a game running at 60 FPS does not mean that you get 16.6ms mouse-to-screen latency at all)
In 16.6ms a modern processor will execute about 128 million instructions. We need to locate about a screenful of text and/or cat pictures. We need to perform layout and then render it. The amount of work being done is not very much, or very difficult. The resources available are luxurious.
What part of the problem is it that makes it an unrealistic target?
Sure, so there will be latency, maybe 4-5 frames with input lag. But this is normally bearable in a game so I'm guessing that it would work fine for an office app. It would be difficult to remove, but 80ms response time from an application that can always hit its 16.6ms deadline should be workable.
Hard disagree. Sorry, but rant inc ...
If we can't hit 60fps with a chat app, one of the simplest applications imaginable by man, what the hell can we hit 60 with? It's drawing a couple thousand triangles (maybe) with a couple hundred textures (I guess). Game engines fart a couple thousand triangles when they're done drawing 50x+ that. Oh, and they do about a gazillion other operations .. at 60fps+
The slow part of switching between conversations is fetching the data which, on a fiber connection in a city, is < 5ms round trip. Hell, I live in the boonies in Canada and my round trip to Seattle is only like 20ms. You could also cache the first 20 messages to every conversation (disk space is cheap) to avoid the network.
As for the notion that "you need to care about such low-level things ...", yes, you do. To write fast software you just deal with it and move on.
Seems like you're simply not aware of complexities of the display path. Please make a simple windowed OpenGL application displaying a single triangle in thousands FPS that changes color when you press a key and measure its latency, you may be surprised ;)
If we're splitting hairs about how we define how long an application takes to update, we've already won.
In reality, we're talking about an app that should take ~0.01s to perform an operation (update a rectangular region with text), that's taking 1.0s instead. And that's THE OPTIMIZED VERSION. In the real world, we've already lost, which is what Casey was pointing out in his tweet.
And yes, of course, something is seriously wrong with a lot of applications these days, otherwise we wouldn't be having this conversation at all.
"It's a bit of understatement to say @WebKit project cares about performance or efficiency of the software. We're quite obsessed with it, and we throw a lot of engineering resources at it, not to regress, and to improve upon."
Then they aren't "engineers" yet. Engineers can figure out how to use a profiler and how to improve performance of their software. Calling themselves "engineers" while being unable to do so should be a signal to everyone in their circle that they're incompetent hacks. They should at least be honest and give themselves an appropriate title, like "apprentice" or "intern" or "novice" or "student".
> A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyse a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.
I think all software engineers should be taught to be able to handle a profiler, poke around in a debugger, write a short correctness proof, and consider how an attacker might abuse the APIs they expose. I know a lot of people who are working to make this dream a reality right now. Until this happens, though, gatekeeping the definition of an engineer is inappropriate and does nothing to help improve the situation, because people are generally resistant to learning these things if you call them "incompetent hacks".
Profilers are simple tools that every software engineer should be familiar with. They are literally just a way to measure performance of a software system. The basic ones do this by tracking time, or counting the number of times a block of code are entered. People have recreated them a million times (probably more) because they're so damned easy to create.
I'll take my hit for this one: If someone were to come up to me and say, "I'm an engineer, but I don't know how to measure the software I make." I'd think they were a hack, yes. If they instead were honest and said, "I'm a student studying to become an engineer ..." or "I'm a novice trying to become better..." then at least they're honest individuals and not liars or hacks. If that's gatekeeping, then so be it. They are not engineers yet if they can't use the software equivalent of a ruler.
It's not gatekeeping.
Traditional engineering fields have somewhat standardized expectations around what a new grad should know. Same as law or Medicine/Nursing. When someone calls himself a Software Engineer, I too have an expectation of what they should know.
It's part of what made the scene where Katherine Johnson explains Euler's method as a little known math trick to a group of bewildered NASA engineers so funny. Everyone in that room would have been familiar with that concept as it's the first numerical method encountered in a differential equation class [0].
I would love some sort of standardized curriculum for Software Engineering. No exam, just a list of pre-requisite and concepts.
[0] The real work she did would have probably required a separate two hour documentary so I understand the filmmaker's decision here.
>> A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyse a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.
“For as soon as the distribution of labour comes into being, each man has a particular, exclusive sphere of activity, which is forced upon him and from which he cannot escape. He is a hunter, a fisherman, a herdsman, or a critical critic, and must remain so if he does not want to lose his means of livelihood; while in communist society, where nobody has one exclusive sphere of activity but each can become accomplished in any branch he wishes, society regulates the general production and thus makes it possible for me to do one thing today and another tomorrow, to hunt in the morning, fish in the afternoon, rear cattle in the evening, criticise after dinner, just as I have a mind, without ever becoming hunter, fisherman, herdsman or critic.” - Karl Marx
Therefore we should hide the truth?
Sorry, it doesn't matter what field you are in, if you can only build stuff but you lack at least a solid familiarity with the tools and techniques of your trade that enable you to improve the performance of your product/service, you are simply not yet fully competent.
Period. Full stop.
If you can write code that compiles and runs, and even do it at large scale, that's nice. But have some humility and recognize that while this is definitely far better than nothing, it is NOT the end, it is merely the beginning, and you are not yet anything special.
Same applies for any field. It is really great go from nothing to being able to build functioning stuff, but it is only the beginning.
Sadly, many people get one year of experience 20 times over, and then get all full of themselves thinking this is the same as 20 years of of digging deep into the technology. Saving them the hard assessment does no one any good.
It's fair to expect a professional cook to be able to cook well. Guarding their feelings for their incompetent cooking serves no purpose, since they become useless to their firm, and customers will want to avoid what they cook.
The same expectation should hold for professional engineers. If they are incompetent hacks, tell them they are. Guarding their feelings above describing reality to them only lets their illusion of competence to continue, and is what causes poor software, more incompetent incompetent engineers, and troubled users.
It is the incompetent hacks who have failed themselves, by not being aware, and by not adapting and improving accordingly. The 'system' did not fail them.
The real world is not a college or school or university, where things are fed to you on your time. Employees must learn on their own time and other people's time. If they cannot keep up, then let's be real: they are incompetent.
By blaming the system, you take away the accountability and responsibility of incompetent people. If many better programmers were able to catch up, why can't they? We can't make excuses for incompetence.
What are they doing?
> Until this happens, though, gatekeeping the definition of an engineer is inappropriate and does nothing to help improve the situation, because people are generally resistant to learning these things if you call them "incompetent hacks".
So they should instead be coddled?
How does it not help to simply say: Here's something everyone from a decent engineering program should know on graduation?
> How does it not help to simply say: Here's something everyone from a decent engineering program should know on graduation?
This does.
> So they should instead be coddled?
This doesn't. I'll let you figure out what the difference is between these two.
They don't?? If that's true, it's a pretty sad commentary on software engineering these days.
> it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.
When a company grows and add features that customers need, the data model has to grow with it. Queries necessarily become more complex.
I used to work at a big video sharing site. I wasn't there in the beginning, but I imagine the initial profile pages had code like
SELECT * FROM user_videos WHERE user_id = ?
So then the simple query became
SELECT * FROM user_videos WHERE user_id = ? AND privacy = ?
And then the query suddenly became much more convoluted. We started loading all of a users' videos in memory, then in-memory filtering, returning only the videos the viewer had access to (slow for users with thousands of videos).
Then some users came along with 30,000 videos and we had to abandon the in-memory filtering because we were getting OOMs, so we developed a hybrid system that could run queries either in-memory or in the DB depending on volume.
This is a trivial example (there are much more painful examples of data model bloat) but it illustrates the point: if you think it should be easy, you probably have an innacurate mental model of what needs to happen behind the scenes.
That's like saying that Chrome is a modern UI for what used to be Lynx. Those tools offer very different features, even if they share a rudimentary set of behaviours.
E.g., clicking the "emoji" window in Slack. I'll permit the custom emojis in the LRU emoji list to delay, but even the stock Unicode "text" emoji take palpable time to "load".
Now think about the average chat app like Slack or Teams. Those have a very different read-write ratio. Some things can be cached for long periods of time, but many other entities become invalidated quickly, and are subject to intricate permissions policies.
As for sharding, Slack uses Vitess: https://slack.engineering/scaling-datastores-at-slack-with-v...
> most people on Twitter aren't writing Tweets
I don't know how true this is. Most people in my corner of twitter tweet several times a day, some of them have rapid-fire conversations in the replies. And that cache-busts (or at least appends to) a whole lot of people's feeds, just like it would people's chat history (while in both cases, almost all writes are appends)
But idk, I'll stop speculating here because I'm not basing it on much
Probably a few thousand features you have never heard of and will never see or use. And the vast majority of their customers will never see or use. But each one has one or two clients for whom someone decided it was vital at the time to implement that feature to please that client.
You can probably imagine one small feature that only one or two clients use that adds a very tiny, almost imperceptible amount of loading time. Now just multiply that by several thousand.
Seems like when you hire bean counters and MBAs, the incentive/value structure of your team/org/company changes for the worse.
It's not just that Teams is failing at solved problems, it's also that they aren't using Microsoft's own SDKs and APIs themselves anymore. Windows has good fast UI frameworks and all that cool stuff like SuperBoost which can make app startup appear instantly. Why is nobody using it?
And there is no useful integration to speak of. Why do I need to re-upload an OneDrive file into Teams? The exact same file is already stored on the exact same server. Does nobody there know about the magic of SHA512?
And how come if the internet has a hiccup while I upload a file, I have to re-upload the whole damn thing? In 1980 we already had resumable uploads via FTP.
Oh and who's the vendor creating one of the best performance tracking and optimization tools? Microsoft themselves. Did nobody ever dare to run this in a proper Visual Studio?
Telegram does that wit just a handful of people.
But yeah I won't blame them not to use xamarin/maui.
When you work in orgs with hundreds or thousands of developers working at a breakneck pace churning out feature after feature for years, on a perpetual "sprint", your architecture tends to resemble a multi-layered bowl of spaghetti. A single button click may result in hundreds of database calls (maybe concurrent, maybe cached), or at a minimum invoke dozens of javascript packages in a Rube Goldberg-esque fashion to perform an action which should take a modern computer a single digit number of milliseconds, but instead results in 10 second load times. Sometimes people just completely forget about entire code paths, and then it becomes dangerous to delete since the persons(s) who wrote it either no longer work there, or got reorged and moved on.
There are absolutely people who care about software performance, but it isn't a priority because it usually doesn't make you more money, won't get you or your boss promoted (unless it saves money), and is difficult to maintain in the long-term given the pace of change in large codebases. When you do tackle performance problems, it's because you were absolutely forced to due to necessity. In certain domains it is also more important e.g. games, embedded systems, ML applications, etc.
One other thought, is that many developers today cannot build software from first principals, as they never learned how. Many only know how to build within the context of a browser environment, and think a "minimal" app is https://create-react-app.dev/ . That's not just me grumbling, it's how our industry works. You forget the layers below you, and build new ones, each generation forgetting most of what the prior knew. The result is giant bloated applications, and a normalization of slowness in software.
I still have hope for small teams of dedicated devs. Fast, coherent software gets written by tiny teams or single individuals.
I too am curious of what sort of... dynamics are going on internally. I blame Agile/Scrum lol.
Sounds like they just rewrote the whole thing from scratch, down to replacing Electron.
This is what you get when your developer team has only ever known programming through the lens of terrible tools and practices. They’ve probably never worked on good software before.
they replaced electron with a MS Edge Web View which is still running V8. I wouldn't say they rewrote it "from scratch"—that would be more like rewriting it in C#
Users don't care because Teams still basically functions and the performance problems are at most a minor inconvenience. As a result, organizations don't care because Teams is cheap and does everything they need it to. As a result, Microsoft doesn't care because dedicating dev resources to something which neither saves them money nor attracts more customers is a waste.
Microsoft employs many smart people and I'm sure Teams' performance could be improved by orders of magnitude if they wanted to. But why would they want to?
(And to reinforce my point, when the performance _did_ get so bad that people apparently actually started caring, Microsoft not only improved it but made a video advertising it. It's just that there's no incentive to optimize any further than is necessary.)
In my experience, the further up the org chart you are, the less likely you are to use Slack directly, instead deferring to assistants or at least only posting announcements periodically.
Ditto goes for Internal IT who I've commonly found never use these chat platforms instead deferring (correctly) to ticketing systems.
Those two groups are both the ones who would probably advocate for adopting these platforms yet they never use it in any regular sense so they don't feel the sharp edges
Presumably, Teams was so unbelievably slow that it crossed over to being an actual problem for customers and not just a mild nuisance. So Microsoft improved performance just enough to stop affecting sales. Is the performance _good_? No, it's still comically slow. But there's simply no incentive to improve the performance further.
But:
> there's simply no incentive to improve the performance further
The breakdown here (https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...) indicates this was a monumental rewrite, almost green-field, with the singular goal of fixing performance. I don't think they slumped on effort this time
People definitely care. If you actually listen to them. They will tell you what a hot pile of slow garbage it is.
Architecture suffers, code health, and performance suffers. Silly things are done, but it's hard to see they're silly because it's hard to see the entire picture
Could you build it better from scratch rather then organically? Of course. And often that's part of the solution to these situations.
tl;dr Building software is hard, and its difficulty does not scale linearly with complexity. (And yes, Teams/Slack are complex, even if they "shouldn't be")
...I can't write CSS and nobody moans so why should web devs be expected to engineer software?
Maybe modern software is like that. Just a hundred small inefficiencies that don't look like much when benchmarked on an individual part of the stack but combine into a monster.
One way to test this...MSFT could open the Teams API and some 16 year old could try building a client.
I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
I've built high-complexity, high-performance web apps, and while there are challenges (the smooth infinite scroll without any placeholders in the video was actually pretty slick), there are other things like 900ms to switch tabs and 9.1s to boot that are just really hard to believe (and those are the improved numbers!)
Some of it's probably back-end; Slack for sure has back-end performance problems, where the UI is snappy but then sits there and loads the conversation for several seconds (or has synchronization problems, or loads an old version of history, or...). And I can understand having some performance challenge here- these apps deal with huge amounts of data per-workspace. But also... it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.
I'm just so fascinated. The HN commentary around this topic is usually lazy, but there is a genuine mystery here.
Edit: A light overview of the new version was posted below (https://news.ycombinator.com/item?id=35352521) but I'm linking it again here for visibility: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...