God I love the Not Invented Here disease. Somebody, somewhere, thought "hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman' so lets write our own chat application.... how hard can it be?"
Boom, now the company is straddled by a shitty, homebrew chat application that is maintained by nobody because the original author left for greener pastures. Nobody dares replace it though because that would be sacrilege. That chat app is part of the company, after all!
Arg.... I hate, hate, hate when engineers with a bad understanding of business and too much time on their hands lock their organisations into homebrew crap that has nothing to do with the value delivered by the business.
/rant
Like I completely agree with your overall point about reckless overengineering, but you also just listed three true and really good reasons to avoid integrating Slack.
No reason to use Slack when such options exist, but also no reason to write your own chat app, either.
When OP says they built their own internal chat app, that's not entirely accurate - uChat is built upon Mattermost.
Mattermost has been rock solid for the last couple of years for my team BTW.
This is valid not just for Mattermost but many other apps that support sending e-mails. And that's why it's important that instead of giving up and outsourcing it, we all try to set it up and have first-hand experience in challenges we have to face in the process. Because if everybody just uses Mailgun & SES, in a decade or two it won't be be possible to set up your e-mail server anymore (of course you will be able to, but your e-mails won't reach most customers).
Unfortunately, I found this to already be somewhat of a reality. Even with all the best practices I went to, many MTAs simply don't trust my IP address. Most "sane" MTAs seem to greylist me rather than blacklist outright, so it just adds a bit of delay to users receiving their emails. Thankfully it seems to be gradually getting better, but gone are the days of simply pointing your apps at sendmail letting them go.
I ended up outsourcing DMARC report processing, if I wanted to do it in-house I'd have to setup and maintain an Elastic stack[1] which seems like huge overkill.
You'll find a lot of times the champion of the new hotness moves on and leaves someone else to maintain the old hotness at costs that were never factored into the original deployment.
As I noted in another comment, it's a trilemma where you have to pick and choose your battles. The initial deployment time is rarely the most important factor when introducing a new stack within an organization.
For example, large MMO guilds have a quite different regulatory environment from, say, a large healthcare organization in the US governed by HIPAA. It's an extreme example, sure, but a homegrown, dumbed down solution, might be the only way to ensure regulatory compliance.
It doesn't take a lot of imagination to come up with other scenarios. I can't defend Uber doing it because I'm not involved, but large MMO guilds are not archetypal of enterprise systems.
Adding to other large company problems, these people will need management layers, the team will need a charter for growth. Who wants to be the maintainer of the Mattermost servers in a large company? Add all of this up and then the slack deal starts to look reasonable.
Edit: Just to add some real numbers, slack costs $12.50 per user per month [1] - that is ~$750k per year for 5000 users = 5 engineers. (More like 3 really in the Bay area)
There's no way a company with 5000 Slack users is paying the sticker price per head - they'll call Slack's enterprise sales team and negotiate some volume discount with whatever shiny enterprise-grade features, SLAs, support guarantees, etc. they need.
There's no way it suddenly needs a new team of dedicated full time resources.
If it takes you that many engineers you are either using the wrong software or using it incorrectly.
Moreover, email is hard and does require a dedicated team, but the order of magnitude is wrong here. 5 people is plenty.
These Bay Area tech people must be either incredibly inefficient (especially when comparing to what they get paid), or your estimate must be waaaaay off.
Compare with a small development team for a significant percentage of their time to build a halfway decent chat tool.
Miles of difference.
> Slack began as an internal tool for Stewart Butterfield's company Tiny Speck during the development of Glitch, a defunct online game.[13][14] "Slack" is an acronym for "Searchable Log of All Conversation and Knowledge.
His first game company also didn't do well, so they pivoted to release Flickr.
Given that Slack can be seen as IRC with some pretty decoration glued up to an Elasticsearch instance, and that Uber already has a highly trained tech org in place, if it wants to make a Slack "clone" for less than, say, $25million (cost of Slack to Uber for 5 years), it could actually be a pretty sensible decision- especially when you take into account the tax and stock-price benefits of capital expenditure.
Maybe they realised it was a superior solution to building their own app.
Not directly related to mattermost and Uber, but companies tend to list clients even if the product is not widely used or was even dumped.
You are a small team in a big company, you evaluate a product and find that it satisfies some of your basic needs but want to work with it for a while, so you purchase a couple of licenses for you team but stop using it after a year or two.
voilà ! you are a client listed on their web site
Uber's internal chat tool is built on top of Mattermost.
But even if they didn't, it's not as simple either as "can those engineers clone this software for less than X", it's whether they can afford to take the opportunity cost of not having engineers work on something core to Uber's business.
Anyway, as another commenter pointed out, on an enterprise scale I'm sure you can get a good deal. I can also imagine however that at that scale you'll want multiple Slack servers / instances - which would add cost given how slack charges per person, per server.
So there's a few alternatives then; self-hosted (some parties will offer enterprise customers the option to host an instance of their application on the customers' internal hardware), or rebuild it yourself. The latter one however takes a bit of math - how much is it going to cost to build, maintain and run it? It's easily underestimated.
And yet, it's also easy for people working in an engineering culture with nearly limitless engineering capacity to just build it themselves - motivations being "how hard can it be", and "I need something more challenging than tweaking this button for my employer to earn 0.1% more money on this particular feature". I've seen it happen far too often.
None, they're all independent contractors.
Even the public kubernetes workspace has almost 75k people in a single channel.
What what does scale mean anyway? Just employees using it or employees + contractors?
It's easy to armchair quarterback on this stuff, but I find it hilarious how tech companies that employee supposedly smart people justify writing this kind of stuff. I'm sure some junior engineer did a cost/benefit that completely neglected the total cost of ownership for writing a chat app and focused on the sticker shock of having to pay tens of thousands a month for a chat app.
I've worked in plenty of companies with engineers who think they are legal, privacy and security experts. If we made business decisions based on these folks, we'd never do anything. Hell, every damn text change we makes invokes a few who worry about the legal ramifications ("should we file a JIRA ticket with legal?").
Sadly without somebody stepping in, people actually listen to them instead of the actual experts employed by the company... thus you wind up wasting tons of engineering bandwidth building a feature / platform / product that has no business existing all (or worse, not building some feature / platform / product) because nobody bothered to check with the legal / privacy / security team that some concern by a engineer was actually a genuine concern.
And even then, in any good organization legal / privacy / security teams exist only to point out all the risks in any business decision... sometimes it is worth the risk despite what legal / privacy / security say. What appetite the company has for the risk depends on a lot of things that are well outside of legal/privacy/security.
the general counsel and thus the entire legal org required for any company approved comms system a way to automatically by default destroy chat logs after 7 days (and to turn that off for folks who were on legal holds). slack in 2016 did not have that capability.
Booo on legal.... like I said earlier, it's easy to armchair quarterback. Sometimes I wonder how the fuck humanity even manages to make progress with so much waste and boneheaded decision making.
Running your own DC is the same as building your own chat app. It is almost always more expensive and almost all the people who claim they can do it cheaper inhouse aren't factoring in all the costs--both costs that are easy to measure and those that are almost impossible to measure
Examples of hard-to-quantify costs for running your own infrastructure include:
- What is the cost to the company of being three versions behind on mongo because IT doesn't have the bandwidth to upgrade? How many work-arounds do teams have to do because they couldn't use the latest features of mongo?
- What is the cost to the company when developers cannot quickly spin up new environments like they could using a service like heroku?
- How much innovation and new business (read $$$$) is not happening because the in-house IT simply doesn't have the toolset so a dev can quickly riff on an idea?
- How many engineers have good ideas and don't bother building them because spinning up an environment is too much trouble?
I'd argue by not using AWS you are holding your product teams back and stifle innovation. Unless of course you want to waste company money to build out your own half-assed provisioning tools that are lousy knock-offs of AWS's tools. But then, you might as well just use AWS...
What I was trying to point out is someone engineered a system that costs 8M/mo to run yet only handles 1M rides per day.
That doesn't really sound that bad, considering that rides can easily cost dozens of dollars. Just about any other non-tech business would kill for such low overhead.
Other non-tech businesses, hell EVERY business that I've ever worked in that used AWS did better per transaction - healthcare, digital advertising, virtual office tooling, gaming. If you can't do better than a credit card processing fee per paid customer interaction across the company on AWS, you might as well be selling retail goods...and depending on age, your company might be soon.
Companies like Lyft are doing a lot more than updating a few fields for each ride. They would be processing massive amounts of location data and they are building out models to eventually use for self driving cars.
It is sizable, not ridiculous. And optimizing your cloud cost is definitely your own responsibility and deserves every bit of attention.
Since when did wanting to be a software developer mean there’s “one true way” to do software dev?
Why are developers supposed to be budget babysitters for a huge company with tons of people watching budgets?
I don’t owe my employer being an expert in all contexts to the benefit of their business. I am not their servant. I write code to serve contexts they want served.
Which is even more scary. Why isn't uber laying all these people off and paying cash money for a real chat application? What competitive advantage does uber have maintaining a chat application?
You can take your hands off completely, even go and do something else, read a book, write an email, book some flights, and the chat will drive itself ... somewhere.
What's it called this week?
[1] https://en.wikipedia.org/wiki/Trade_dress
[2] https://revisionlegal.com/trademarks/lessons-trademarking-tr...
I am talking about Samsung and Apple here..
Except Snapchat, I still can't work out what that's about.
Look at their competitors in Asia (all-in-one services like WeChat, GoJek etc. where search, chat, ride hailing, banking etc. are all built in one app) and it would make sense for them to have an engineering team building a chat app.
https://news.ycombinator.com/item?id=17623005
I have the impression that Slack has gotten worse. We're again having problems of notifications not showing up.
Yes, really, for business. Look past the theming.
Discord has a much more solid tech base (and engineering team) behind it than Slack, IMHO. A better API, too; and—unlike Slack—real support for third-party clients!
I built my own niche team-collaboration app back in 2012, and I tried out a lot of things, but ended up building my whole own stack. That was back then. If I was doing it again today? I'd just make it an alternative Discord frontend.
(There are also offerings specifically designed for the "build your own custom frontend for our chat backend" use-case, like https://pusher.com/chatkit, but IMHO Discord even has these beat.)
You don't have to make those links (and you can turn off, per server, the ability to create those links.) You can invite specific users. The user already has to have a Discord account, though.
And that's the thing; Discord has a different accounts model. Which is part of what I was referring to when I said they had a more solid tech base: when you're signed into a dozen Slack workspaces, the Slack client has to keep a dozen websockets open, because the open connection is for a given Slack account's view of the world, and Slack accounts are a sub-resource of Slack teams. This isn't a scalable model, and when the Slack app is using 1.2GB of memory and two full CPU cores, this is much of the reason why. When you're signed into a Discord account, on the other hand, you just have one connection to the server—and one state-sync session that includes all your open channels in all those servers—because your profile on a given Discord server is a sub-resource of your global Discord account.
It's the difference between how an email client connects to N accounts through IMAP, and how the Gmail mobile app is connected to N GSuite accounts.
And, as such, it wouldn't really make sense to have Discord SSO at the account level, because a Discord account is associated with the individual, not with a particular employee record of a particular Enterprise. It's not like an email address; it's more like a Keybase identity or a Facebook profile. (You could certainly have particular profiles under the Discord account that did SSO to a particular server, but this wouldn't be traditional SSO-as-we-know-it; it'd be more of a "connector" flow, like when an app wants to use your Github profile.)
Github is a good comparison, actually. Most corporations just use Github, not Github EE. You don't Enterprise-SSO to Github.com. Github.com is an SSO provider, in fact, that other sites (e.g. Asana) can take auth from!
As for github.com, our official IT recommendation that people don’t use their personal accounts, but create one just for work. This way there is no worries about personal data there.
(This is at a subsidiary of a large international company. I can believe small startups use a simpler methods)
Keep in mind, putting SSO into Discord’s account system (if that ever happened) wouldn’t be like putting Enterprise MDM on your personal phone. It would be more like having Chrome Sync signed into both your personal Gmail account and your GSuite account, as separate Chrome “People” (or whatever those are called.)
With Chrome Sync, the profiles are still isolated from one another; but updates to them ride in the same sync carrier connection, because that connection is going to Google either way. In this sense, the Chrome installation itself, and its “installation-specific sync client ID”, is like a Discord account. It’s not a personal account, or an enterprise account; it’s a nothing in-and-of-itself, that has those types of objects under it.
[1] https://github.com/Microsoft/vscode/wiki/Performance-Issues
[2] https://github.com/jhallen/joes-sandbox/tree/master/editor-p...
I'm quite fine with that, frankly. Highlighting correctly requires actually parsing the file. I use vim and the colors get wonky way too many times, because it tries to "keep it fast".
Slack has decided that each team workspace must be firewalled off from the others: such that it needs its own websocket, its own client-side sync session, its own open channels, etc. This is because Slack was first-and-foremost a web-app, and in the Slack web-app, you’d just have one browser tab per Slack team. Slack built up its infra assuming this “one tab per workspace” model, and it crept into their backend API design, and now they’re kind of stuck with it. So now, even in the native mobile app (let alone the desktop Electron app), when you have 12 Slack teams open, you have basically 12 copies of the app’s view-controllers open and idling in the background, 12 background sync controllers, etc. In the case of the Electron client, that’s also 12 renderer processes, just as if you had 12 literal browser tabs open!
You might not notice this at first, because Slack won’t initialize all those resource for a workspace’s “tab”-equivalent until you actually focus the given workspace at least once. But if you just skim through all those workspaces once in the morning, and leave Slack open, it’ll be hogging those resources all day.
(And, on mobile, that’s especially a concern, because despite what you say, web-sockets themselves have heartbeats, and 12x the heartbeats is 12x the reasons for your phone’s modem to wake up. If you’re ever wondering why the part of your phone around the antenna is getting hot even when you’re not actively using data—it’s probably that you have several active Slack workspaces in a live Slack app-container.)
Don’t put words in my mouth. A heartbeat is not high resource usage.
Your wall of text doesn’t negate the fact that multiple websockets can be maintained with trivial resource consumption. It absolutely cannot be generalized to assume that multiple websockets means copies of everything. I’ve seen multiple clients supporting multiple websocket connections to distinct servers while re-using all of the same UI components.
Bridging in Slack, Gitter, IRC and other platforms makes working with external teams a breeze, everything is in one place.
Perhaps the memo didn't make it around to everyone.
The equal and opposite problem is managers who don't understand the sunk cost fallacy and refuse to throw out home grown solutions. Sometimes there might have been a good idea to homebrew a thing years ago because nothing good existed. Now several better solutions exist but the company refuses to adopt them because doing so would mean "throwing away" years of work, and the managers who signed off on that work are afraid they'll look bad.
Apparently there was a serious suggestion that this new www was shoddy and we needed to write our own version of http - this got stamped on by some senior people at Martlesham (UK version of bell labs)
Well that's the incentive for that kind of behaviour, pad your resume and get another job.
> hey, slack is way to expensive, we don't want to get 'locked in', and 'mumble mumble privacy cloud evil stallman'
gets followed up immediately (almost inevitably, in my experience) with
> so lets write our own chat application.... how hard can it be?"
...rather than, say, "hey, I googled and there are a few FOSS alternatives for 'group chat with history': Zulip, for one; Mattermost, for another. Let's see if either of those fit our needs. If they don't quite, though, let's also see if either one of them has a solid, customizable codebase to fix up!"
And even then they could have just used Matrix, which ticks all of those boxes.
The real issue here is a lack of vision at the top. The current CEO needs to step down for the health of the company. He has a clear lack of vision for the company, as evidenced by his statement of "do your divisions look like what they should if we had to start over?" Who says that? Something so vague that is basically grasping at straws and shows a complete lack of strategy. Would Jeff Bezos say something like that? I highly doubt it, because he has complete control over his company and a vision that has worked at micro and macro levels.
You can do things that aren't directly related to your core business and these things should generate value both internally and externally. If you can't do this, eventually, your company will stop growing.
This where understanding business matters. AWS makes sense as a business and it benefits mostly from economies of scale which is something unique that a massive company like Amazon can provide the service which few other companies could.
NIH stuff is also rarely externalized outside of the business. Even as open source. Which as the OP pointed out needs proper investment of human capital. Not some side project of a bored engineer or two.
Most innovation happens outside of mega companies by small teams for a good reason. Even the Skunkworks type approach is only useful for certain types of work like Googles X projects. Some B2B SaaS programs roll out of parent organiations but typically they are started by teams who quit the bigger company to solve the problem they had.
The first AWS service was launched over 15 years after Amazon.com started selling books on the Internet. It wasn’t even a separate brand until after Amazon S3, SQS and EC2 were launched.
Close. It was roughly 11 years. I've been using Amazon AWS since early 2007; I believe it launched to the public in the form we know today in early 2006. They started selling books online to the public in mid 1995.
If you go back to the very first AWS services - 2002 - it's closer to being only seven years.
If a different company had dominated the on-demand computing space before Amazon’s peak load issues became too big a problem, they very may as well have gone with that and not made AWS.
Almost nobody needs to write their own chat app. Slack is fine.
RMS was always right especially with regards to privacy.
Look familiar? https://mattermost.com/
That's interesting, I interviewed with Uber for an engineering management role in 2016, and was blown away when my potential boss said he had a "hiring target" for next year of close to 100 people. This was a relatively small product offshoot of Uber that is now closed. I thought that was absolutely nuts, not just the magnitude of the number but that getting bodies in seats was a metric that he/I would be measured against, but thought maybe that I just didn't understand how hypergrowth startups work and it would be a good way to gain a new perspective!
Full Disclosure: I did not get an offer, even though I walked out of that interview feeling I had never nailed an interview as much as I had that day.
There was some earlier discussion here on why trust between employees and employers is so low now, and this is a great example. Employees being laid off while new positions open up for that exact same role so that they can hire new developers at a lower cost or avoid having to promote other employees.
I often wonder how accurate the roadmap prioritization can be. I do agree it’s pretty silly to build a lot of infrastructure that will never effect your clients.
If you have for instance a billing process that needs to be automated, custom infrastructure is needed. Manual approvals of all transactions is a lot even for banks. Operations can always benefit from something that cuts their overhead time down a significant chunk. Operations approval on transactions seems like an operations problem but clients having to wait for approval means now the problem is shifted.
I assume you're not on the side that got laid off. I wonder what their opinion is...
>I think the lesson here is to grow responsibily, and it's real people's lives that are being affected. Don't hire people to alleviate your current engineer's burnout and stress.
Because those are trivial matters, and they should just suffer through them?
Seems like infrastructure/devops always gets short-staffed, presumably due to being considered a cost center. Interesting to hear someone observe this even as they're saying a company "over hired" and "engineers ran out of things to do".
Disclaimer: I work on the M3DB team.
Obviously everything I say is personal experience and opinion. I think the layoffs were long overdue and should've happened sooner. There was a general understanding between my coworkers and I that Uber definitely over hired in 2016. Thanks to that a lot of engineers ran out of things to do which led to political infighting over roadmap, ridiculous redundant resourcing - every single team had mobile/web, severe shortage on infra teams since head count was already taken by main Uber teams. We even started building and maintaining our own chat app. I disagree that all the cool eng project that we put out and share here is a waste of time or resources. Those project was initiated by real needs on Uber and only the best make it out into the rest of the world. Engineers that shipped those projects are still here and we would've done it regardless of whether the 435 people that were hired in the first spot. I fully realize that it's an insensitive thing to say but I think it's important that it's expressed somewhere.
I think the lesson here is to grow responsibily, and it's real people's lives that are being affected. Don't hire people to alleviate your current engineer's burnout and stress. Learn to plan better and to prioritize aggressively instead of just hiring and getting everything done.