News for hackers. Stuff that matters.
Re:Uber lays off 435 people by 22c on Tuesday September 10, 2019 @11:12PM (#20936253)
I setup an enterprise-grade Mattermost server in a weekend. The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers.

No reason to use Slack when such options exist, but also no reason to write your own chat app, either.

Re:Uber lays off 435 people by IanPBann on Wednesday September 11, 2019 @07:29AM (#20938812)
Uber have talked about their internal chat app, uChat.

When OP says they built their own internal chat app, that's not entirely accurate - uChat is built upon Mattermost.

https://eng.uber.com/uchat/

Re:Uber lays off 435 people by webXL on Wednesday September 11, 2019 @10:30AM (#20940574)
That kinda sucks the air out of this whole thread. "that's not entirely accurate" is a bit of an understatement. But, don't let facts get in the way of a good rant!

Mattermost has been rock solid for the last couple of years for my team BTW.

Re:Uber lays off 435 people by dvfjsdhgfv on Wednesday September 11, 2019 @02:35AM (#20937171)
> The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers.

This is valid not just for Mattermost but many other apps that support sending e-mails. And that's why it's important that instead of giving up and outsourcing it, we all try to set it up and have first-hand experience in challenges we have to face in the process. Because if everybody just uses Mailgun & SES, in a decade or two it won't be be possible to set up your e-mail server anymore (of course you will be able to, but your e-mails won't reach most customers).

Re:Uber lays off 435 people by 22c on Wednesday September 11, 2019 @06:41AM (#20938511)
> of course you will be able to, but your e-mails won't reach most customers

Unfortunately, I found this to already be somewhat of a reality. Even with all the best practices I went to, many MTAs simply don't trust my IP address. Most "sane" MTAs seem to greylist me rather than blacklist outright, so it just adds a bit of delay to users receiving their emails. Thankfully it seems to be gradually getting better, but gone are the days of simply pointing your apps at sendmail letting them go.

Re:Uber lays off 435 people by 22c on Wednesday September 11, 2019 @06:36AM (#20938473)
What's interesting is setting up a handler for DMARC aggregated feedback reports. I determined that it's not worth doing yourself, even Microsoft do not handle their own DMARC reports.

I ended up outsourcing DMARC report processing, if I wanted to do it in-house I'd have to setup and maintain an Elastic stack[1] which seems like huge overkill.

[1] https://domainaware.github.io/parsedmarc/

Re:Uber lays off 435 people by jholloway7 on Wednesday September 11, 2019 @02:49AM (#20937243)
Let's not pretend that hosting an OSS chat server is a walk in the park for every enterprise just because you personally got one up and running in a weekend.
Re:Uber lays off 435 people by Piskvorrr on Wednesday September 11, 2019 @03:45AM (#20937523)
Let's not pretend that building one from scratch isn't at least an order of magnitude harder, either.
Re:Uber lays off 435 people by jholloway7 on Wednesday September 11, 2019 @10:43AM (#20940706)
I'm not necessarily in the 'build one from scratch' camp either, I'm just pointing out that there's a trilemma in enterprise systems. How fast you're able to get something up and running is just one factor and not always the most important one.
Re:Uber lays off 435 people by mucholove on Wednesday September 11, 2019 @03:58AM (#20937581)
Looking at this from Spain, you did do this over the weekend. I also work 6 days a week—but my hope is that this didn’t take the place of building a family, having a relationship, finding a relationship going for a hike in the mountains, citizen science, or making art. Your weekend is valuable!
Re:Uber lays off 435 people by a_t48 on Wednesday September 11, 2019 @05:07AM (#20937934)
You're totally right that the weekend is valuable, but some people enjoy doing this sort of thing with their free time. It's not anyone's place to decide what anyone else does for entertainment or self improvement. That is to say - what is not valuable to you might be valuable to someone else.
[dead]
Re:Uber lays off 435 people by a_t48 on Wednesday September 11, 2019 @01:12PM (#20942425)
No, that pretty much falls under "None of your business".
Re:Uber lays off 435 people by danjac on Wednesday September 11, 2019 @08:56AM (#20939552)
Why? If you're not a friend or family member, it's none of your business.
Re:Uber lays off 435 people by jholloway7 on Wednesday September 11, 2019 @11:04AM (#20940923)
My comment isn't concerned with using 'personal' time for such endeavors. My concern is with the implicit conclusion that how fast you can get something up and running (a 'weekend' in this case) is an argument that you can and should deploy a new stack in any given enterprise.

You'll find a lot of times the champion of the new hotness moves on and leaves someone else to maintain the old hotness at costs that were never factored into the original deployment.

As I noted in another comment, it's a trilemma where you have to pick and choose your battles. The initial deployment time is rarely the most important factor when introducing a new stack within an organization.

Re:Uber lays off 435 people by Ntrails on Wednesday September 11, 2019 @05:24AM (#20938005)
When large MMO guilds have had a mostly trouble free chat client that supports a few thousand concurrent users on the free time and good will of admins I'm loathe to believe that this shit is impossible
Re:Uber lays off 435 people by jholloway7 on Wednesday September 11, 2019 @11:51AM (#20941471)
I'm not sure how you read from my comment that I'm saying 'this shit is impossible'. I'm simply saying in many organizations 'how fast the intern got it up and running' is not always a valid indicator of how ready it is to be deployed to a few thousand concurrent users.

For example, large MMO guilds have a quite different regulatory environment from, say, a large healthcare organization in the US governed by HIPAA. It's an extreme example, sure, but a homegrown, dumbed down solution, might be the only way to ensure regulatory compliance.

It doesn't take a lot of imagination to come up with other scenarios. I can't defend Uber doing it because I'm not involved, but large MMO guilds are not archetypal of enterprise systems.

Re:Uber lays off 435 people by mirkules on Wednesday September 11, 2019 @05:44AM (#20938101)
The funny thing about your comment is that Slack was born out of a MMO game.
Re:Uber lays off 435 people by raz32dust on Wednesday September 11, 2019 @01:17AM (#20936780)
If the company relies on Mattermost, I would assume they'd need at least one maintainer per ~100 users, just to make sure that the SMTP keeps working reliably, there are backups etc. One person to maintain Mattermost would cost a company at least 150000 USD in the bay area. That's $100+ per month per user. Slack is way cheaper than that.
Re:Uber lays off 435 people by dx034 on Wednesday September 11, 2019 @01:45AM (#20936924)
Really, 50 maintainers for a 5000 user system? I'd think it scales much better, probably 2-3 people for the whole company. And in that case, it can quickly be cheaper than slack
Re:Uber lays off 435 people by amerine on Wednesday September 11, 2019 @02:04AM (#20937011)
Exactly. No way does it take 50 people to scales mattermost to 5k because of smtp. This op made that all up.
Re:Uber lays off 435 people by raz32dust on Wednesday September 11, 2019 @03:12AM (#20937366)
I think you are really underestimating the bureaucratic overheads in a large company. It is not the same as a 3-person startup serving 5000 customers. Of course I pulled that number out of the air, but the point is that it can make sense. I think I am right around the ballpark of needing a 5 people team to serve 500 users - you'd need a minimum of 3 people just to make sure there is enough redundancy if one person is sick or goes on vacation. You'd need 24x7 on-calls. The scaling factor probably changes after that - may be it is 10 people for 5000 users instead of 50.

Adding to other large company problems, these people will need management layers, the team will need a charter for growth. Who wants to be the maintainer of the Mattermost servers in a large company? Add all of this up and then the slack deal starts to look reasonable.

Edit: Just to add some real numbers, slack costs $12.50 per user per month [1] - that is ~$750k per year for 5000 users = 5 engineers. (More like 3 really in the Bay area)

[1] https://slack.com/pricing

Re:Uber lays off 435 people by candu on Wednesday September 11, 2019 @08:26AM (#20939269)
Your general point is solid, but one quibble with the numbers - even the pricing page has an "enterprise" tier "for very large businesses".

There's no way a company with 5000 Slack users is paying the sticker price per head - they'll call Slack's enterprise sales team and negotiate some volume discount with whatever shiny enterprise-grade features, SLAs, support guarantees, etc. they need.

Re:Uber lays off 435 people by sciurus on Wednesday September 11, 2019 @06:45AM (#20938535)
Or you have your existing corporate IT team manage it the same way they're managing other internal apps.

There's no way it suddenly needs a new team of dedicated full time resources.

Re:Uber lays off 435 people by linuxdude314 on Wednesday September 11, 2019 @02:00AM (#20936995)
This is a more realistic scenario. I own/operate shared services for thousands of users on a team of 3.

If it takes you that many engineers you are either using the wrong software or using it incorrectly.

Re:Uber lays off 435 people by catwell on Wednesday September 11, 2019 @04:02AM (#20937602)
At Uber scale, you probably already have your own SMTP servers used by all your employees, so that's not really a problem.

Moreover, email is hard and does require a dedicated team, but the order of magnitude is wrong here. 5 people is plenty.

Re:Uber lays off 435 people by kortilla on Wednesday September 11, 2019 @02:46AM (#20937225)
Lol, where did you get the idea that smtp servers need more maintenance as users increase? What if I told you I’ve witnessed a team of 3 manage a mail system of a university of over 100k active email accounts?
Re:Uber lays off 435 people by Slartie on Wednesday September 11, 2019 @06:25AM (#20938378)
We are a company with around 100-150 employees, using Mattermost in a real Enterprise setting for years now (with SSO, CI and mail integration and whatnot) and that server is being maintained by our admin team basically "on the side" - nobody was hired specifically for this job, as it does take less than 20% of one person's time according to what the team says.

These Bay Area tech people must be either incredibly inefficient (especially when comparing to what they get paid), or your estimate must be waaaaay off.

Re:Uber lays off 435 people by jholloway7 on Wednesday September 11, 2019 @03:05AM (#20937335)
You're making a good underlying point but contriving way too many details in your argument for it to be taken seriously.