striglia

I find this infra perspective so fascinating, as a career-long product platform engineer and solidly staff+. I don't argue that sean's writing (which is amazing) is a little overly mercenary in framing, but I'll pick a few choice sections I don't agree with here too. tldr I think the "product cares about speed; infra cares about leverage" staff engineer archetype is a false dichotomy we shouldn't encourage.

"In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts." -- I disagree that speed is the ultimate currency! A great product org also respects long term leverage, it's just _always_ harder to argue for. But it's best to build a strong portfolio of going fast (where needed), going slow (where high leverage), and if everyone agrees your "going slow" led to huge returns you get the best of all worlds. Frankly it's a sign of a relatively junior product engineer if they are myopically focused on speed at the cost of all else.

"But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons." -- Extremely true, and this is _equally or more true_ in product domains. My most valuable contributions have come from sitting in a product area long enough to generalize 5 micro optimizations into the macro engineering leverage we needed to drive an order of magnitude more value from the same engineering input.

"For some engineers, navigating this [high visibility driven] chaos is a thrill. For those of us who care about system stability, it feels like a trap." -- my protip for prospective staff engineers is to _never_ say you only care about [speed, stability]. In most cases you must care about both, and it's worth advertising yourself as such. If you self select out of companies that only care about [pure stability/pure ship velocity] there should be a valuable balance to strike and staff engineers are in a unique place to enshrine that balance in engineered systems.

"In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers." -- surely we can agree impressing customer stakeholders is even more important in (healthy) product orgs :) But it's a curious claim!

Great article, wonderful to hear more nuanced and deep discussion of the practice of extremely senior IC engineering. Kudos!

Finally, I cannot wait! This sure sounds like it should remove the terrifying guesswork of wondering what the actual quota values are, detecting when you're approaching a limit, and starting to automate or at least alleviate the pain of changing them.

Judging just by the blog post, this might well be the best quality of life improvement of 2019.

In the console, I see at least some services (I care about Step Functions) list as "showing default quotas only". I wonder if this means there will be a long tail of adoption before individual services show actual current values?

Strong agree -- the distribution was a solid deal when it happened and didn't require a paywall. I just gave up on them (http://www.locallyoptimal.com/blog/2019/03/24/publish-indepe...) now that they're forcing you to choose no-paywall or medium distribution. If I'm not wanting a paywall I may as well use my own site and be totally free.

It's a shame, I found the editing interface pretty convenient (image embedding in particular) and conceptually the Medium-run publications make a ton of sense.

Inherited a codebase that ran nearly all revenue-critical operations, and operated at its core on some of the most metaclassy/dynamic tools Python has at its disposal.

Luckily I work in a tech company where the fact that nobody knew this code and it took years to effectively ramp up on was argument enough that it should go away.

Actual removal was a much messier story. It had tangled deeply into adjacent systems so you couldn't "just replace it". We are in the later phases of something like the Strangler Pattern (https://docs.microsoft.com/en-us/azure/architecture/patterns...) where we built higher-level interfaces over the top and gradually re-implemented the underlying functionality without using any of these custom frameworks.

That said, it's a long term project that is easy to lose steam on. It's been very important to regularly revisit our goals and how we're attacking them...AWS has released services that fundamentally changed our approach (for the better) in the years since this effort started and we've probably cut off at least a year from the overall effort by adopting those instead of continuing on the original course.

I wrote up some of these ideas about accomplishing big projects that span years at https://medium.com/@scott_triglia/ask-a-tech-lead-i-have-to-.... The parts about regularly re-evaluating the next steps in your course of action were directly inspired by this project I just described.

Great to see open source tooling around Step Functions. Personally I'm still holding out for someone taking the States language spec (https://states-language.net/spec.html) and turning out higher-level tooling for writing/inspecting/manipulating state machines.

Agree w/ bpicolo (hi Ben!) that the tooling still can come a long way for those who can't use Lambdas directly. This code is interesting, but it misses my major use case for Step Functions -- running a workflow across several services transparently.

If I had to guess the future, I suspect that tools like Step Functions will become more and more crucial as we realize that in a FaaS world the job of coordination is complex and essential.

Interesting video from Tim Bray showing off the variety of Step Function uses in the wild https://www.youtube.com/watch?v=sMaqd5J69Ns. Lots of Lambda, but not solely!

What makes you say Step Functions isn't ready for prime time? We've been using it (and SWF which it is based off) for about a year at decent scale and been generally very happy with it.

Exactly -- and I guess my question is whether there is any reason to believe they won't succeed :(

So I ask this mostly out of horrified curiosity -- is there any negative press or revelation that could actually sink Uber at this point? What would it take for the company to fail?

We've seen:

* Systematic flouting of laws in public

* Even further hidden avoiding of laws

* Sexism complaints across the board

* What appears to be a pervasive toxic culture, inspired (if not explicitly encouraged) straight from the CEO's behavior

Are they actually too big to fail?

Very much looking forward to an upcoming project with Step Functions at $DAYJOB. I suspect there are many use cases for turning complicated intersections of batch jobs and blobs of code into discrete, independent tasks like SFN requires.

Particularly interested in composing higher and lower level state machines for better separation of concerns, but haven't yet come up with a compelling use case :)

Is it possible to accept an activity task with no timeout, store the state in a DB, then when your event occurs, load state from the DB and send the task_success signal? Not convenient, but seems feasible..

As someone currently at the "prove it's a good idea" stage of moving production from SWF to SFN, I'm happy to hear it's been such a good experience in practice!

I think I'm mostly excited to stop locally representing/maintaining/enforcing the state machine -- it's been a huge source of toil for very little benefit. Hoping the cheapness of a new state machine will encourage splitting them when it makes sense instead of inflating/complicating an existing workflow...

How are the latency gains in practice? Specifically the activity -> decision -> activity interstitial latency in SFN vs the old SWF style.

By far the most frustrating part of blogging for me (though certainly not the only one) is the assumption that no one will ever see it. I can probably solve this by options I dislike -- giving up ownership to Medium or obnoxiously cross-posting to all social media platforms -- but the end result is I am discouraged before I even start.

Oddly I love speaking at conferences...maybe because some nonzero audience is more or less guaranteed?

Cool article. I really like repetition that model complexity is not a pancea. Seems like the industrial AI/ML movement as a whole has gone down a road where practitioners will, by default, throw the most powerful model they know at a problem and see how it pans out. Works well on benchmarks(if you regularize/validate carefully) but isn't a very sustainable way to engineer a system.

Separately, I do find it curious that his list of "pretty standard machine-learning methods" included Logistic Regression, K-means and....deep neural nets? Sure they're white hot in terms of popularity and the experts have done astounding things, but unless I've missed some major improvements in their off-the-shelf usability they strike me as out of place in this list.

This is exactly what the posted article doesn't do though, and I really appreciate that. In fact it's absurdly rare to see someone share their success story while owning up to the fact that it 1) may not be universal 2) was not planned out in advance 3) may not even be repeatable. Major credit to Danilo for not falling into that trap.

I think you underestimate the importance of context in all of this. You can't build any of these products well without knowing (deeply and certainly) users' patterns. Microsoft has no realistic way of knowing when it routes you to the wrong place...no continuous training data to let it feed and refine a personal assistant. Data products can be cool demos (see Siri) from a startup, but without both the starting dataset and the continuous feedback that you get by deploying these against real users and capturing the resulting behavior, I think they are quite limited. Continuous adaptation and learning is as important as the initial algorithm.

And while I certainly am looking into the future when I talk about cornering the market, I think it's an interesting question to ask how you compete against a company like Google that starts with this much user context. How can you build better experiences when they have the advantage in data, ML expertise, and the raw computational power? The incremental acquisitions of data and companies are not the issue...it's the web of connections that is valuable. And Google is not only trouncing Apple, their only meaningful competitor, in that race...Google is pulling even further ahead.

You're comparing them on uneven terms. Show me one company besides Google that can even stack up to what Apple's done in Maps or with Siri. There's none. That's because both require amazing amount of expertise, money, and data to deploy properly. Apple has correctly done what they had to (break off the data flow from iOS to Google) even where it may have reduced the experience for their customers. That's a long term play.

Funny how Google is continuing to corner the data market via acquisitions like Nest. Almost feels...anti-competitive, no? I wonder how long until we start talking about Google the way we still talk about 1990s Microsoft.

Is it plausible that a new company, starting from scratch in the Maps/Search/Personal Assistant/etc. field could actually compete with Google? Or would they just get crushed under the personalizing power of Google's raw data and computational advantage?

When I started working in SF (moving from the Los Angeles area) I had great luck renting a temporary place and apartment hunting after work hours. I used AirBnB but you would be fine with any approach...the point is that apartment hunting is an infinitely better experience in person and you can avoid all the craigslist horror stories involved with remote renting.

At the same time, I think it's worth noting that a disturbing amount of the criticism made within the programming sphere is not aimed at solving a problem, but rather exposing the obvious wrongness of the original product and its author(s).

I fully support the idea of discussing a problem, but the key term here is discussion. Sitting around talking about how terrible component Foo is because it's obvious to even the most casual observer that it has not a single redeeming characteristic is strictly demonizing.

For me, it's all about the community. Recognize that every program we criticize almost certainly had someone pour a ton of effort and thought into it, and use that knowledge to temper your criticism. Make it a learning experience for the original author, and strengthen your team/organization while you're at it instead of tearing down people in the name of improving the project.

Most of the locations here are probably sourced directly from their data providers they list, including Acxiom and Factual. At this scale, they probably haven't geocoded each place themselves.

What experience I've had with those data sources makes me unsurprised that there are misplaced businesses.

Love the presentation of the various API calls in each language. My only nitpick is one I have for a lot of code displays like this -- selecting the code also selects line numbers =)

Google has spent a long time pouring a ton of money and manpower into their geocoder and the data behind it. It's completely unsurprising that Apple is not able to match their quality. It will be an iterative process that improves as they get more usage data to leverage.

That said, your example is pretty shocking considering Apple's historically stringent quality standards.

This seems like a useless statement. Surely no one is consciously failing people they know fit the team and can perform the job. The whole point of interviews is noisily assessing these things. Point me at a reliable, cost-efficient way of determining that and I'll gladly champion it.

Always cool to see articles trying to tease apart correlation from causation. That said, I am automatically suspicious of any publications mentioning power laws, since they make for attractive stories and can easily be fit to data which is not actually generated from a power law. See http://cscs.umich.edu/~crshalizi/weblog/491.html for an excellent discussion.