News for hackers. Stuff that matters.
Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @01:24PM (#22392445)
>It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case.

Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed. Whether or not the work we were undertaking was meritious and valuable is an entire branch of philosophy I'm pretty sure.

Part of the struggle at big companies is that a lot of existing solutions just don't work. Let me use an example with chat. A few years ago Slack was evaluated as a replacement for HipChat, since Atlassian's outages had finally started affecting us during our own outages.

Everybody wanted to go to Slack, but the cost of Slack was tremendously prohibitive and the state of the service then (as I was told) was such that it could not support a company of Uber's size. Tremendous effort would have been undertaken by Slack to support Uber and they didn't want to expend that effort for a single customer. This was late 2015 early 2016.

There were tons of options, but ultimately an in-house chat software was created. At the time it seemed required to make our own highly reliable chat, considering how distributed engineering teams were. I think if you talk to anybody without the background of how chat evolved at Uber they would think the in-house chat project would have been a boondoggle.

Not all over-scoped engineering projects are actually so noble. There was certainly a ton of "reinventing wheels" going on. There was significantly more "these problems are really hard and I only have bad solutions."

Though if the result is ultimately, "nothing more than interesting repos amongst a sea of interesting repos" sign me up.

Re:M3DB, a distributed timeseries database by jjeaff on Saturday February 22, 2020 @09:04PM (#22394612)
There seems to be a pervasive misconception by the very employees at Uber, that they built their own chat platform. When in reality, and someone please correct me if I am wrong, uChat was a white labeled Mattermost.

I have heard that the team that put it together actually tried to hide that fact from the company (for the glory, I guess). But that could be apocryphal.

Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @09:28PM (#22394697)
That's entirely not true. The team was always forthcoming about the fact it was Mattermost, at least to other engineers.

Mattermost didn't work out of the box, and certainly not the way and at the scale Uber needed it to. I'm not overly familiar with the technical details, but one thing in particular stands out as an example. There was a Town Hall channel that every user had to be a member of. This unfortunately did not scale, and not enough ACLs were available to limit all the ways users could use this universal room. Eventually they really fixed the problem, but it was a tremendous pain point for a long time. There were a lot of fundamentally "less than great" things about Mattermost that had to get updated to work for Uber.

There was the amusing time employees found out anybody could change the topic in the room, even if our chat permissions had been disabled. It was absolute chaos for at least an hour, I can't remember if it actually negatively impacted the deployment though it sounds vaguely familiar.

It's pretty telling of employees that badmouth the uChat team. That team ultimately was trying to do what they thought was best for the company, even if at the time it seemed like they bit off more than they could chew. There was no other engineering team so directly visible and exposed to the entire company internally like they were. People dismissive of their efforts are generally not used to the difficulty of making so many very vocal customers happy all at the same time, and could be more sympathetic.

Re:M3DB, a distributed timeseries database by jjeaff on Monday February 24, 2020 @03:16AM (#22402469)
I dug up the comment I saw that mentioned this. They also claim to be an Uber employee.

https://news.ycombinator.com/item?id=19101617

The fact that uChat is commonly mentioned by Uber employees as being the "custom chat solution we built in-house". It leads me to believe that comment that the team tried to hide that it was built on open source.

I don't doubt for a second that scaling Mattermost for a huge organization like Uber was a big undertaking. But it seems disingenuous for people to always mention that Uber built uChat when it should be more like "Uber put a lot of work into Mattermost to scale it up."

Re:M3DB, a distributed timeseries database by iamleppert on Sunday February 23, 2020 @02:30AM (#22395624)
Why not just use IRC?? It scaled for the entire internet.
Re:M3DB, a distributed timeseries database by cerberusss on Monday February 24, 2020 @02:42AM (#22402339)
These IRC replacements do more than that. There are voice and video calls, integrated file sharing, and a whole bunch of tools that you can add.

To be honest, I don't much like Slack because I feel the desktop app doesn't feel like a real macOS app. And I don't use all these features. So in the end, IRC would be fine for me. But it wouldn't for the rest of the company.

Re:M3DB, a distributed timeseries database by remote_phone on Saturday February 22, 2020 @01:51PM (#22392585)
Uber is in the process of ditching uChat and moving to Slack
Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @02:15PM (#22392725)
I accidentally left this point out. In retrospect it's easy to say Uber made the wrong decision to make uChat but it was one of few options at the time.
Re:M3DB, a distributed timeseries database by hitekker on Saturday February 22, 2020 @03:30PM (#22393142)
Seems like a huge point to leave out.

Are you affiliated with Uber?

Re:M3DB, a distributed timeseries database by pc86 on Saturday February 22, 2020 @04:11PM (#22393344)
Well the comment starts with "former Uber engineer here" so I'd venture yes but not anymore.
Re:M3DB, a distributed timeseries database by jjeaff on Saturday February 22, 2020 @09:05PM (#22394618)
Doesn't seem relevant unless the point about slack being too expensive and unable to handle the load is untrue.
Re:M3DB, a distributed timeseries database by creddit on Saturday February 22, 2020 @02:31PM (#22392815)
> There were tons of options, but ultimately an in-house chat software was created.

You drank too much kool-aid. uChat was just a reskinned Mattermost.

Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @02:35PM (#22392839)
I think you're being overly dismissive of how much work that team did.