News for hackers. Stuff that matters.
Re:M3DB, a distributed timeseries database by missosoup on Saturday February 22, 2020 @11:25AM (#22391780)
I agree with everything you say.

But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos.

The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that resembles their own operating model. Maybe this project will set a new trend, but so far nothing they put out gained any traction or became suitable for general purpose production use. In that regard, H3 and their other projects have remained at the level of decent 'show HN' pieces rather than something you'd ever use professionally. In other words, marketing.

Based on previous news coming out e.g. https://news.ycombinator.com/item?id=20931644

It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case.

Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @01:24PM (#22392445)
>It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case.

Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed. Whether or not the work we were undertaking was meritious and valuable is an entire branch of philosophy I'm pretty sure.

Part of the struggle at big companies is that a lot of existing solutions just don't work. Let me use an example with chat. A few years ago Slack was evaluated as a replacement for HipChat, since Atlassian's outages had finally started affecting us during our own outages.

Everybody wanted to go to Slack, but the cost of Slack was tremendously prohibitive and the state of the service then (as I was told) was such that it could not support a company of Uber's size. Tremendous effort would have been undertaken by Slack to support Uber and they didn't want to expend that effort for a single customer. This was late 2015 early 2016.

There were tons of options, but ultimately an in-house chat software was created. At the time it seemed required to make our own highly reliable chat, considering how distributed engineering teams were. I think if you talk to anybody without the background of how chat evolved at Uber they would think the in-house chat project would have been a boondoggle.

Not all over-scoped engineering projects are actually so noble. There was certainly a ton of "reinventing wheels" going on. There was significantly more "these problems are really hard and I only have bad solutions."

Though if the result is ultimately, "nothing more than interesting repos amongst a sea of interesting repos" sign me up.

Re:M3DB, a distributed timeseries database by jjeaff on Saturday February 22, 2020 @09:04PM (#22394612)
There seems to be a pervasive misconception by the very employees at Uber, that they built their own chat platform. When in reality, and someone please correct me if I am wrong, uChat was a white labeled Mattermost.

I have heard that the team that put it together actually tried to hide that fact from the company (for the glory, I guess). But that could be apocryphal.

Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @09:28PM (#22394697)
That's entirely not true. The team was always forthcoming about the fact it was Mattermost, at least to other engineers.

Mattermost didn't work out of the box, and certainly not the way and at the scale Uber needed it to. I'm not overly familiar with the technical details, but one thing in particular stands out as an example. There was a Town Hall channel that every user had to be a member of. This unfortunately did not scale, and not enough ACLs were available to limit all the ways users could use this universal room. Eventually they really fixed the problem, but it was a tremendous pain point for a long time. There were a lot of fundamentally "less than great" things about Mattermost that had to get updated to work for Uber.

There was the amusing time employees found out anybody could change the topic in the room, even if our chat permissions had been disabled. It was absolute chaos for at least an hour, I can't remember if it actually negatively impacted the deployment though it sounds vaguely familiar.

It's pretty telling of employees that badmouth the uChat team. That team ultimately was trying to do what they thought was best for the company, even if at the time it seemed like they bit off more than they could chew. There was no other engineering team so directly visible and exposed to the entire company internally like they were. People dismissive of their efforts are generally not used to the difficulty of making so many very vocal customers happy all at the same time, and could be more sympathetic.

Re:M3DB, a distributed timeseries database by jjeaff on Monday February 24, 2020 @03:16AM (#22402469)
I dug up the comment I saw that mentioned this. They also claim to be an Uber employee.

https://news.ycombinator.com/item?id=19101617

The fact that uChat is commonly mentioned by Uber employees as being the "custom chat solution we built in-house". It leads me to believe that comment that the team tried to hide that it was built on open source.

I don't doubt for a second that scaling Mattermost for a huge organization like Uber was a big undertaking. But it seems disingenuous for people to always mention that Uber built uChat when it should be more like "Uber put a lot of work into Mattermost to scale it up."

Re:M3DB, a distributed timeseries database by iamleppert on Sunday February 23, 2020 @02:30AM (#22395624)
Why not just use IRC?? It scaled for the entire internet.
Re:M3DB, a distributed timeseries database by cerberusss on Monday February 24, 2020 @02:42AM (#22402339)
These IRC replacements do more than that. There are voice and video calls, integrated file sharing, and a whole bunch of tools that you can add.

To be honest, I don't much like Slack because I feel the desktop app doesn't feel like a real macOS app. And I don't use all these features. So in the end, IRC would be fine for me. But it wouldn't for the rest of the company.

Re:M3DB, a distributed timeseries database by remote_phone on Saturday February 22, 2020 @01:51PM (#22392585)
Uber is in the process of ditching uChat and moving to Slack
Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @02:15PM (#22392725)
I accidentally left this point out. In retrospect it's easy to say Uber made the wrong decision to make uChat but it was one of few options at the time.
Re:M3DB, a distributed timeseries database by hitekker on Saturday February 22, 2020 @03:30PM (#22393142)
Seems like a huge point to leave out.

Are you affiliated with Uber?

Re:M3DB, a distributed timeseries database by pc86 on Saturday February 22, 2020 @04:11PM (#22393344)
Well the comment starts with "former Uber engineer here" so I'd venture yes but not anymore.
Re:M3DB, a distributed timeseries database by jjeaff on Saturday February 22, 2020 @09:05PM (#22394618)
Doesn't seem relevant unless the point about slack being too expensive and unable to handle the load is untrue.
Re:M3DB, a distributed timeseries database by creddit on Saturday February 22, 2020 @02:31PM (#22392815)
> There were tons of options, but ultimately an in-house chat software was created.

You drank too much kool-aid. uChat was just a reskinned Mattermost.

Re:M3DB, a distributed timeseries database by carlisle_ on Saturday February 22, 2020 @02:35PM (#22392839)
I think you're being overly dismissive of how much work that team did.
Re:M3DB, a distributed timeseries database by Scarbutt on Saturday February 22, 2020 @01:32PM (#22392488)
Ignore the project and move on? If you expect every open source project to be served by all your entitlements, you will be repeatedly disappointed.
Re:M3DB, a distributed timeseries database by lazaroclapp on Saturday February 22, 2020 @02:02PM (#22392644)
> The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that resembles their own operating model.

Not a contradiction. Many of these tools are suitable for use in production, almost by definition, since they are being used in production, at Uber. They might or might not work in your environment out of the box. But they are certainly often likely a better starting point than an empty editor, even when they do not. Most of the ones I am familiar with, are happy to get PRs generalizing them to more varied environments.

> they're nothing more than interesting repos amongst a sea of interesting repos.

As someone who has open-sourced on GitHub: research prototypes hacked together for a research paper deadline in grad school, class projects, for-fun hacks, and also production tooling I built as a paid engineer, I'd say there is a big difference! :) And there would still be a big difference even if the later were somehow never touched again after the first "we are open-sourcing this!" commit.

That said, we do try to maintain the things we open-source. Standards of support vary because individuals maintaining these projects, and their situations, vary. This is true for non-OSS internal tools too. In my experience, having gone through the Uber OSS process twice, and having started it a third time and decided against releasing (yet?), Uber does try to make reasonably sure that it's open-sourcing stuff that will be useful and is planned to be maintained. At the same time, they have to balance it with making it easy to open-source tools, otherwise too many useful things would remain internal only.

Also, note, some of these tools have exactly one developer internally as the maintainer, and not even as their full time job. For example, I am the sole internal maintainer[1] for https://github.com/uber/NullAway and also have 3-4 other projects internally on my plate, most of which are in earlier stages and need more frequent attention[2]. If and when said developer leaves, effort is made to find a new owner. This is not always successful, particularly if the tool has become non-critical internally. Sometimes, leaving owners retain admin rights on the repos and keep working on the tool (Manu, NullAway's original author, co-maintains it), but I don't think anyone is suggesting that that should be an obligation.

Finally, obviously, nothing here is the official Uber position on anything, just my own personal observations. This doesn't represent my employer, and so on. I am also pretty sure most of this is not even Uber specific :)

[1] Not the only internal contributor! Also, there is one external maintainer, as mentioned a few sentences later. But in terms of this being anyone's actual responsibility...

[2] Just to clarify, I think between Manu's interest, my own, and it being relatively critical tooling at Uber, NullAway is pretty well maintained. But I can understand why that isn't always a given for all projects.

Re:M3DB, a distributed timeseries database by mamon on Saturday February 22, 2020 @12:19PM (#22392101)
Isn't that kind of the point of open-sourcing your internal tools? You don't want to be bothered with maintenance and support, so you're hoping that some anonymous volounteers will do that for you :)
Re:M3DB, a distributed timeseries database by lopsidedBrain on Saturday February 22, 2020 @01:16PM (#22392397)
Pretty much every successful open source project that people pay attention to is one that has had long-term support behind it. Linux, Mozilla, gcc, clang, git. Almost always, that support begins with the original author.

Protects that don't do that are therefore unlikely to remain interesting for long.

Re:M3DB, a distributed timeseries database by cfors on Saturday February 22, 2020 @01:04PM (#22392336)
Maybe I’m a cynic of all big corp companies but if you’ve ever worked with a big corp open source department that’s almost the entire point is to build PR for the engineering department. Same goes for tech blogs. These things will be PR pieces first, and valid production tools/frameworks second (mostly).
Re:M3DB, a distributed timeseries database by closeparen on Saturday February 22, 2020 @01:27PM (#22392459)
In my experience software gets written in the first place for the usual internal reasons. Corporate or individual prestige may be the driving factor in open sourcing, though, rather than a genuine interest in having it used externally.