ecoffey

Would love to hear more, since I largely used SLOs on backend services (which in turn called other services that also had their own SLOs).

As far as timespans for the error budget consumption, I’ve seen 1 hour -> 1 day -> 1 week. The 1 hour error budget rate would be a page and the others would be low priority.

So you could either keep that as the alerting and/or use the error budget “look ahead” to see if there are more specific alerts you need.

I certainly agree in spirit that the alerts are important, and should be actionable. But I wouldn't start at just "looking at the service" and then trying to define the first set of alerts.

Instead I would move up a level and start with a SLO for the various "business level" metrics you might care about. Things like "request latency", "successful requests", etc.

Then use the longer lookahead "error budget" burndowns to see where your error budget is being spent, and from there decide 1.) if the SLO needs adjusting, and/or 2.) if an alert is appropriate.

To cleanly answer those questions and iterate you'll need metrics, dashboards, traces, and logs. So then you're not just making dashboards because "its best practice", you're creating them to specifically help you measure if you're meeting your stated service objectives.

https://sre.google/sre-book/service-level-objectives/

Interesting! Reading the headline before the article, my brain immediately thought of "jitter".

I wonder if you could extend the `In-process synchronization` example so that when `CompleteableFuture.supplyAsync()` thunk first does a random sleep (where the sleep time is bounded by an informed value based on the expensive query execution time), then it checks the cache again, and only if the cache is still empty does it proceed with the rest of the example code.

That way you (stochastically) get some of the benefits of distributed locking w/o actually having to do distributed locking.

Of course that only works if you are ok adding in a bit of extra latency (which should be ok; you're already on the non-hot path), and that there still may be more than 1 query issued to fill the cache.

Northguard doesn’t look like it’s been open sourced? I’d be curious to know how it compares to Apache Pulsar [0]. I feel like I see some similarities reading the LI blog post.

0: https://pulsar.apache.org/

That is tough, I’m sorry for your loss.

Even loose coupling is still coupling. For the things that have to be coupled having the code organized in the same place, being able to easily read the source for “the other side”, make a change and verify that dependees test still pass, etc is immensely powerful.

In my experience microservices are easier to manage and understand when organized in a monorepo.

Monorepo is one of few things I’ve drunk the koolaid on. I joke that the only thing worse than being in a monorepo, is not being in one.

> perfect ASMR for nerds.

Haha yeah I like to have this on it the background when I’m doing other things.

Bartosz links to it in the Further Reading section, but wanted to highlight the Wristwatch Revival YouTube channel[0] as well. Really great content and very understandable after reading the article!

0: https://www.youtube.com/c/WristwatchRevival/videos

> LSP optimizes writing the code.

I would actually phrase that as “LSP optimizes for understanding” (which is of course important for writing code).

For example, when doing code reviews I routinely pull the branch down and look at the diff in context of the rest of the code: “this function changed, who calls it?”, “what other tests are in this file?”, etc. An IDE/LSP is a powerful tool for understanding what is happening in a codebase regardless of author.

Makes me think of this project: https://www.vesta.earth/

Trying to see from specs and such if the mesh supports ethernet backhaul vs wifi radio like the google wifi does.

There is a great book called The Emerald Mile that touches on some of this history, and other Grand Canyon history, using the story of the fastest traversal as the narrative device. Highly recommend. https://www.goodreads.com/book/show/15803144-the-emerald-mil...

re: implicits... https://twitter.com/ecoffey/status/1179198362924404736?s=21

last year we got solar installed (8kW system + 10kWh battery): https://twitter.com/ecoffey/status/1212399375986487296

replaced a minivan with a leaf: https://twitter.com/ecoffey/status/1036815557008318465

I bike to work: https://twitter.com/ecoffey/status/1208223257695940611

We've been talking about converting a chunk of lawn into a bigger garden as well.

You can shift click a range, or command click a disjoint set, of tabs in chrome. And then with one mouse drag move all of them to an existing window or create a new one. It's a feature that I wish safari had (although I have tried tab expose)

You can still have the question. But my answer is a random 32 character string of alphanumerics :)

A linked list of levels reminds me of Binomial Heaps

Reading that man page sounds like the way to go about it is start with `tame(0)` as early as possible in the program and see how it fails. then iterate pushing `tame()` down and expanding the flags until the program functions.

I don't know :) titles are inherently political. They don't exist in some platonic vacuum. So some part of the progression is about broadcasting the status change to the larger organization. Beyond that I think the ladder becomes about company wide cross-cutting concerns, enough success to be an internal rallying flag. Things like that.

But I've never been in the position to either be promoted or do the promoting to that level :)

While Management usually the only thing equated with "soft" (hate that word beeteedubs) skills, moving up the IC ladder also requires good inter-personal skills.

Above senior you are allowed and expected to go "heads down" (by yourself or a small group) and knock out solutions to technical problems. However you are also expected to identify cross-cutting problems, rally the right people, coherently frame your arguments, etc to move the "implementing solutions to technical problems" needle (even if you don't directly write a lot of that code).

Management also get's the "fun" things like "let's talk salary!" or "you need to improve along X, Y, Z axis!" :P

Just my two cents :)

Great point. Different scale needs different things for sure. Even at the scale that I've found this successful in you still have to consider that backward/forward compatibility, albeit at a much coarser grain.

> because each component was released and versioned separately.

For SOA a mono-repo with synchronized releases is a huge help.

In one commit you can refactor a shared module and update all apps that depend on it. And you know when you deploy to staging or prod that all machines got the latest code

> That's assuming that things haven't become deeply coupled within the "call stack".

Yep; Monolith or SOA you still have to write SOLID code :)

> Separating things across a socket often forces a separation of concerns, which does tend to make refactoring easier.

It /should/, and if it does now your changing more than the one variable of "invoked across stacks -> invoked across socket", and are more likely to introduce a regression :)

Yes, but discovering and refactoring to those clear APIs is easier across the in-process call stack. Have to delay the "hoist and introduce a socket" until you're /reasonably/ confident you have the API more-or-less settled

For me the big takeaway is this:

Refactoring across the call stack is orders of magnitude easier than refactoring across a socket.

Sacrificial or not, you can still write the Monolith as "Service Oriented", just that boundary is the call stack. Especially if you're comfortable with IoC and DI.

Building on the latter I've had success with that. Stubbing out hardcoded concepts that I know will come from a as-yet-unwritten service in the future. Then you start pushing those hardcoded things "down and out"; e.g. it was hardcoded in App A, but now App A requests it from App B, but App B just has the hardcoded thing.