News for hackers. Stuff that matters.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by uberemployee on Sunday September 13, 2020 @04:22PM (#24463676)
When I worked at Uber, this project was openly mocked. One of the CTO’s biggest failures was implementing a promotion scheme where you needed to create a new service in order to be considered “innovative”. This promotion scheme marked what I consider the end of Uber’s engineering excellence and the start of what made Uber turn into a bureaucratic mess.

One of the VP’s of engineering called it “toil vs talent”. People who “toiled” at work, meaning doing good maintenance work, would be rewarded with good bonuses but those with “talent” would be rewarded with promotions. Of course this drove people to come up with fake new services so that they could demonstrate “talent”. This also lead to an explosion of new services that overlapped or did nothing useful. Instead of working together, groups would make new services instead of working with existing service-owners because they needed to justify writing a new service. It was sickeningly transparent.

This project was one of those projects. It has no real use case because why the fuck would we want to use GPUs except to look cool on your resume. The sad thing is that the projects is overstating how well it’s being used internally. Internally people use Pinot instead of this.

For all you future CTOs, consider your incentive schemes carefully and don’t be so far removed from the action that you can’t see when your org is rotting. This is what the CTO did, and like I said, it was one of his biggest failures because it gutted the engineering org. Instead of working together, every team was looking at get promotions at the expense of the company and it showed.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by PragmaticPulp on Sunday September 13, 2020 @06:28PM (#24464463)
I wish I had understood this earlier. My past company made a push to hire from top companies like Uber for some key positions. Some of them were great people who were relieved to be out of the FAANG rat race. Others were single-mindedly focused on rewriting everything they touched with cleverly-branded project names, regardless of whether or not it made business sense.

Early I on, I made a harmless comment in Slack about how one person’s pet project wasn’t a good fit for our needs so our team would be using the older, more proven solution. Later that evening the person pulled me aside, almost in tears, begging me to never say anything critical about his project in a Slack channel again. He explained that at his previous role, success or failure depends entirely on the perception of one’s personal projects and that seemingly innocent comments could tank someone’s promotion chances for years.

I felt bad for him because he had clearly come out of a toxic situation. However, one of his teammates later warned me that he was keeping a journal of potentially incriminating things that I had said in Slack and a detailed log of every issue that he could find with our team’s project in case he “had to use it against me later”.

I could never tell if this was a unique experience or the norm at some companies like Uber.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by uberemployee on Sunday September 13, 2020 @06:47PM (#24464574)
What you describe was definitely not a common occurrence. Engineering wasn’t toxic for many years until the last 9-12 months or so I would say. Pre-Susan Fowler memo, it was the best company I had ever worked at. From 2017-2019 we sort of stalled because of the internal drama and it didn’t get really bad until the last 9 months, where attrition of our best engineers and vile political maneuvering from the dregs made it too much for me to stick around.

The engineer you describe sounds like they have mental health issues. There may be some teams with terrible managers but all companies have this, and I’ve seen similar or worse situations at Amazon.

Most engineers I worked with were great but there were many engineers that “played the game” in order to get a promotion and more money. It was sickening but if that’s the way the CTO sets the incentive scheme, who can blame an engineer for following it? It’s more on the CTO for setting the terrible culture than the engineers.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by kamaal on Monday September 14, 2020 @09:10AM (#24469676)
Being blind to politics rarely ever means politics isn't going on.

Pretty much every company out there has a concept of 'promotion packet', its basically building a case for one's promotion. Of course in a company the budgets are fixed, and so are promotion cycles(yearly in most places). You miss out a turn, you could lose an year, or even risk losing two. In that case its fairly common for managers to build a list of accomplishments(file/packet), and rival managers to build a anti-case/defence for the same. Stack ranking eventually is all about a combination of merit+advocacy+lobbying+counter-lobbying at so many levels that I'd say the engineer who cried wasn't wrong at all.

This is the case in nearly every company. We just wish to delude ourselves that politics is absent at some places.

This sort of power play comes with the territory in a large people structure.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by manigandham on Monday September 14, 2020 @02:30AM (#24467398)
One thing that has been crystal clear in the past few years is that just because someone has the competence for high-level work like software engineering doesn't mean they can't be immature and crazy at the same time.

And these people are louder and more common now because it's easy to hide criticism and claim accomplishments with all the politics, buzzwords and general sensitivity these days.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by dennisgorelik on Sunday September 13, 2020 @10:38PM (#24466210)
> begging me to never say anything critical about his project in a Slack channel again

Because he discourages critical feedback - it leads to 2 major problems:

1) He does not improve [because of lack of feedback].

2) He discourages team around him from improving [because of lack of feedback].

Why not explain that to him (and then if he does not understand - fire him)?

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by WrtCdEvrydy on Sunday September 13, 2020 @06:40PM (#24464533)
> one of his teammates later warned me that he was keeping a journal of potentially incriminating things that I had said in Slack and a detailed log of every issue that he could find with our team’s project in case he “had to use it against me later”.

That is not cool.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by hitekker on Sunday September 13, 2020 @06:37PM (#24464518)
What happened to this person?
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by PragmaticPulp on Sunday September 13, 2020 @07:15PM (#24464746)
His strategies worked at first, at least before he had really delivered anything of value. It started to unravel later when he couldn’t deliver on all (or really any) of his promises.

I think he was operating under the assumption that he could build rapport quickly and then hire a team underneath him to get the work done. That might work at a hyper growth startup that values growth over profit, but we were a mature and profitable company looking to keep headcount reasonable. Most of his plans were so over engineered that they would have take 5-10x the engineers to actually finish on time, so they ended up being half-finished projects that requires constant on-call attention.

That was the tipping point for a lot of us in the old guard. There was an exodus around that time, including myself. It’s not worth fighting those political battles day in and day out while walking on eggshells in every Slack channel.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by hitekker on Sunday September 13, 2020 @07:41PM (#24464896)
Sounds like the engineering higher ups turned a blind eye to his antics.

Good on you for exiting.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by hintymad on Sunday September 13, 2020 @11:16PM (#24466404)
I can concur. Most of Uber's traffic for real-time analytics was served by Uber's Elasticsearch clusters with customized querying layer. My team was also a user of the ES service. The service was sometimes unstable, but in general was fast enough for us, and scalability was never a problem. My team got paged only once from the ES team because we drove a huge spike to the team, and they ended up scanning more than 500 million records per second, for which they had to scramble to scale out. But even at that time, there was no visible degradation on our side, let alone outage. The owning team also presented their pain points, and none of the pains had to do with CPU not being fast enough.

By the way, Uber already had two real-time analytics systems before AresDB. One is the aforementioned ES-based service, and the other is Pinot, which was owned by a Pinot contributor. I was in one of those so-called alignment meetings about using AresDB. Engineers from both the ES service and Pinot were there. It was a disaster. The engineers simply asked what AresDB was supposed to solve, and presented charts over charts to show that computation or lack of join operator was never a problem (because for analytics, data streams can be pre-joined), while efficient IO was. The AresDB team simply repeated that join was important, and parallel computation was critical.

I left the company soon after, so I'm not sure how many critical use cases AresDB has been serving since then. Hopefully they do find some sweet spot to justify the cost of developing such system.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by alquemist on Monday September 14, 2020 @01:45AM (#24467166)
Not a big fan of AresDB because of the glaring IO issues. That being said, pushing 'lack of join operator was never a problem' is textbook Sapir–Whorf. 'We've trained our captive users to never ask a question that requires a join, or use convoluted workarounds if they absolutely must do so, and we are proud of it' just leaves a bad taste. I wouldn't be surprised if Pinot were to add join support in the near future...
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by hintymad on Monday September 14, 2020 @02:54AM (#24467539)
True that join is nice. I was just not sure if real-time join is truly necessary for Uber’s use cases, or AresDB is so much better than a query engine like Presto to justify its effort. Or more fundamentally, is GPU the solution to expensive joins?
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by ambrood on Monday September 14, 2020 @12:50PM (#24472303)
Love the shoutout to Gairos. But man that on-call was a pain in the arse.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by yakaccount4 on Sunday September 13, 2020 @04:51PM (#24463860)
The culture at Uber is weird. In my opinion, there was way too much emphasis on titles. An employee's title is literally in your face in almost every internal site. This creates a lot of situations of bias because every time you get a message on slack, one of the first things you see is their pay grade.

In combination with what you listed above, and the rigid, narrow pay bands, it's no wonder everyone is fishing for constant promotions.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by cellis on Sunday September 13, 2020 @07:09PM (#24464706)
Wow. This is such a good comment. I remember when I was a junior engineer and knew more than many of the "senior" engineers at a job a few years back. It didn't matter at all, as seniority outranked sensibility.

I was thinking about this a bit after reading your comment. Now, I say this as having recently negotiated pretty hard for a "Staff" title, and previously had a "Team Lead" title. I think in certain situations it makes sense to have the title authority to shoot obviously bad engineering problems down, but this has been the exception, rather than the rule, in most of my 12 years of being a software engineer.

So your comment makes me wonder: would it make sense to have a system where everyone would simply be called "Engineer", but allow engineers to vote secretly after having worked with another engineer, on various aspects of their colleague's technical expertise. Engineering Managers or perhaps HR would be aware of the engineering votes, but engineers wouldn't, which would remove much of the implicit bias in engineering meetings. Rather, pay grades would be determined by votes, but no one would ever just "Leave it to Yakaaccount, she's the principal engineer". Everyone would have responsibility to be a solid engineer.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by t- on Monday September 14, 2020 @12:00AM (#24466631)
What you describe reminds me of the peer review system in place at a previous company I worked at. The review period was every 6 months. The way it worked was that as an individual you'd rank (and comment on) everyone you had worked with the previous 6 months based on your perception of how much value they added to the company over those previous 6 months. It could be based on some project they delivered, or them simply always answering your questions about something you needed to know to move your own projects forward, or them participating thoughtfully in company wide discussions. Once everyone had submitted their rankings, an algo would figure out everyone's final rank. The various pay level cutoffs would then be decided by eng management.

It so happens that at that company everyone's title was simply 'Software Engineer'. There was no ego bs so it was a great place to work. I think how helpful you were to others being part of the review is another reason for that. There was a way to see some people's 'rank' by looking at the org chart if you really wanted to. In general swe and senior swe would be under an eng manager, if someone was directly under a director that told you something, and if someone was directly under a vp that also told you something.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by underplank on Sunday September 13, 2020 @11:51PM (#24466587)
I think its a good idea to remove implicit bias where ever you can. But doing that by making a structure that does exist be invisible is dangerous. That hierarchy does exist. Even if it's not in your face. It's human nature and it's a good thing in some situations. I think the better angle on this is to have an open culture where the discussion can happen and everyone listens and asks the questions without fear, consensus is built and those with power (because there is always someone with power) wield it sparingly. This is much harder to get to, but it's more sustainable.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by whoevercares on Sunday September 13, 2020 @07:29PM (#24464814)
If members of the team had trusted each other enough to trust the votes that related to pay grade, you’d probably don’t need the votes already
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by aahortwwy on Monday September 14, 2020 @03:20AM (#24467690)
> So your comment makes me wonder: would it make sense to have a system where everyone would simply be called "Engineer", but allow engineers to vote secretly after having worked with another engineer, on various aspects of their colleague's technical expertise.

I worked at a place that had a system like this. Started great, but turned toxic after a hiring spree.

> Everyone would have responsibility to be a solid engineer.

You would think so, but people with other intentions try to find ways to manipulate the system to achieve various strange goals. They're often successful.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by uber176 on Monday September 14, 2020 @12:13PM (#24471869)
Post-Fowler, HR became very concerned about income inequality. They implemented this by severely compressing the range of compensation at each level. For a year or two, there was no way to meaningfully reward a high performer in her level other than to promote her.

Since mid-2019 or so, the company realized this mistake and the pendulum swung back towards TK’s more Ayn Randian compensation philosophy. But the damage was done. FWIW I never saw much of a power dynamic around title. But it is definitely the most important task of an engineering manager to secure promotion-worthy projects and hand them out intelligently. Since promotion is based on impact, it’s almost entirely determined by the project charter. Difficulty or skill deployed in execution is a tertiary concern.

Also given the precipitous drop in equity value, promotions with hefty raises are required just to keep people’s TC in moderate decline instead of freefall.

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by totetsu on Monday September 14, 2020 @04:01AM (#24467863)
I whish this kind of shop talk was part of more sci-fi world's world building. "No-one liked the P-class light patrol missile gunships, they were only commissioned because the space engineering devision was incentivised to deliver new projects..." on second thoughts zzzz
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by heliodor on Sunday September 13, 2020 @05:07PM (#24463954)
I've been wondering why the UI/UX is broken in so many ways that would be easy to fix while the engineers are busy engineering for the sake of engineering. This project, the time series database, etc. Meanwhile, the Android app can't do basic drawing of a car and a moving line on a map, for example. Your explanation of the dynamics explains the state of things quite well!
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by einpoklum on Wednesday September 16, 2020 @06:55AM (#24491379)
> because why the fuck would we want to use GPUs except to look cool on your resume.

Well, there are reasons, but it's true that GPUs in DBMSes is still something that's very immature.

> Internally people use Pinot instead of this.

Can you elaborate a bit regarding the choice of Pinot over other analytic DBMSes?

Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by eezurr on Sunday September 13, 2020 @11:28PM (#24466474)
This reminds of what Eddie Lampert did to Sears. He changed the company's internal structure such that internal teams has to compete against each other for bonuses and relevancy. There was a great write up that I cant find about his strategy and why it spectacularly failed.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by WrtCdEvrydy on Sunday September 13, 2020 @11:49PM (#24466576)
Management theory now tells you to be competitive over small things. If you're doing competitiveness, do it over lunch or something else.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by ethanbond on Monday September 14, 2020 @08:21AM (#24469225)
Honestly mind blowing to me that someone could get to the CTO level of any organization - never mind something of the scale/pay grade of Uber - and think that incentive structure was going to yield anything desirable.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by exhaze on Monday September 14, 2020 @01:12AM (#24466972)
Hey, at least you didn't have to use TChannel and Hyperbahn ;)
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by igmorv on Monday September 14, 2020 @03:06AM (#24467605)
oh my god.
Re:AresDB: Uber's GPU-Powered Open Source, Real-Time Analyti... by person_of_color on Sunday September 13, 2020 @10:15PM (#24466081)
Where did you land after Uber?