jroseattle
Organizations usually suck at this part. Being time & resource constrained, the focus on anything but actual problem solving is just staggering to me. I haven't worked at every place, so your mileage may vary.
It looks like most of the work of the package is reactive -- monitoring external things and responding. It's not proactive just because a person didn't dial it up.
But, here's the thing -- this can likely replace many executives right now, at least those who are reactive managers. I'm in full support -- automate those things if you can.
AI is coming for judgment jobs. It seems it's turtles all the way down.
My mother-in-law lives in rural Missouri and had her 6x10 bathroom remodeled in 2025. Full gut replacement of everything. Total cost: $5600. I got a quote for something similar in a bit larger bathroom in our house in Seattle. Quoted price: $31,000.
Labor is a good chunk of cost, and the OP presented that he simply didn't have the time available to him to do that work himself. On the other hand, I built 20-ish feet of fencing for our backyard this summer, matching an existing design we had done several years ago. Quoted price from the same company that did it the first time: $9800. I did the work myself. My total cost: $1400.
That works out great, right? I also have a significant investment in all the right tools: compound mitre saw, construction-grade table saw, nail guns, impact/hammer/regular drills, angle grinders, power sanders, every drill bit in captivity -- I don't really lack for anything. These make a huge difference in the workmanship of the final outcome.
And that was just a fence. I've considered building an external office here at our house. When I start to take into account the scope of converting an outdoor shed to an acceptable backyard office, I hit the brakes. Electrical power under proper permits, adequate security, something that holds up under our weather, potential water/sewer access, and sufficient space to work (I can't live without a whiteboard)...there is more scope than meets the eye.
All things considered, I think the OP did a pretty good job here in terms of cost control and recognizing what he could do (windows) vs. needed someone else to do (proper electrical.)
> The server is not the most important thing, and each new kind of server that exists in the system is an appreciable increase in maintenance burden.
Sure, but compared to what? It's right to consider complexity, but understanding tradeoffs requires depth. The OP's original premise was that learning all the things about specific queuing services was unnecessary chafe; that INSERTs, SELECTs and UPDATEs are all anyone needs.
The maintenance burden sits with your producers & consumers (or publishers/subscribers, whatever your nomenclature...), whether you want it there or not.
My learned experience is that as soon as you start moving messages that are beyond trivial and carry different operational characteristics, you're going to have to understand those deeper concepts anyway.
In tech stack choices, I prefer staying simple as long as feasible. That said, you also need to know and understand concepts at a thorough level.
The above comment from the article suggests PG as a central server that simplifies event architecture. As if the "annoying, custom and needed skillset" around those specific alternatives are unnecessary baggage.
If you know anything about queues, scaling, availability, access semantics, message formats and concepts such as delivery guarantees, you find out very quickly that the server which stores a queued message is not the high-order bit in that equation.
I don't think the issue was the communications medium (Zoom.)
> I don't think much about my time at Better but it was fine.
That's the biggest tell about leadership at any company.
Maybe we were lucky in our implementation, but a key factor for our decision was understanding how to manage the systems in our environment. We would have skilled up with Redis, but we felt our Postgres solution would be a good first step. We just haven't had a need to go to a second step yet.
50000?
> You need distributed locking if the transactions can take seconds or hours, and the machines involved can fail while they hold the lock. From my experience, locks are needed to ensure synchronized access to resources. Distributed locks are a form of that isolation being held across computing processes, as opposed to the mutex example provided.
And while our implementation definitively did not use a distributed lock, we could still see those machines fail.
I fail to understand why a distributed lock is needed for anything due to it's duration.
Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a request if one is available, assign some metadata from the request to the allocated ticket, and remove it from consideration for future client requests.
We had failed campaigns in the past (over-allocation, under-allocation, duplicate allocation, etc.) so our concern was accuracy. Clients would connect and request a ticket; we wanted to exclusively distribute only the set of tickets available from the pool. If the number of client requests exceeded the number of tickets, the system should protect for that.
We tried Redis, including the naive implementation of getting the lock, checking the lock, doing our thing, releasing the lock. It was ok, but administrative overhead was a lot for us at the time. I'm glad we didn't go that route, though.
We ultimately settled on...Postgres. Our "distributed lock" was just a composite UPDATE statement using some Postgres-specific features. We effectively turned requests into a SET operation, where the database would return either a record that indicated the request was successful, or something that indicated it failed. ACID transactions for the win!
With accuracy solved, we next looked at scale/performance. We didn't need to support millions of requests/sec, but we did have some spikiness thresholds. We were able to optimize read/write db instances within our cluster, and strategically load larger/higher-demand campaigns to allocated systems. We continued to improve on optimization over two years, but not once did we ever have a campaign with ticket distribution failures.
Note: I am not an expert of any kind in distributed-lock technology. I'm just someone who did their homework, focused on the problem to be solved, and found a solution after trying a few things.
One of my personal photos I keep around was taken long ago in what was the biggest bricks/mortar bookseller. I was looking at the selection of books on Java available at the time. O'Reilly was the dominant publisher, and thus had several offerings on the wall. Most of the books were at least 2 inches thick. (If you were ever involved with writing a technical book in the early 2000s, you'll understand the publisher metrics at the time were based on the width of the spine on the shelf.)
Among the many Java manuals of significant girth was a small, THIN book with the title "Java -- the Good Parts". :-{}
Without further information, this sounds like code introduced in a hotfix that wasn't merged back to feature branches.
Surely it's not that simple?
So is interview fraud. The remote-interviewee-answers-questions-while-her-face-reflects-windows-popping-up-on-her-screen is tiring at this point. So, I decided to find a way to inform me if someone was being fed answers in a tech interview.
Behold, the low-tech whiteboard. Also known as a piece of paper and a pencil. With the candidates I've run into that do not pass the "smell" test -- where I think they are being fed answers -- I ask them to draw some things, on paper. It's not a true validation, but it gives me something of a clue.
I ask for a simple diagram. Different services in a network, for example. Or a mini-architecture. For their level, I'll ask for something that should be drop-dead easy.
I ask them to show me their drawing.
The responses I've received run the gamut of "I don't know" (after 5 seconds of deliberation) to "I don't understand the purpose" (after 5 minutes of silence) to "I need to shut off my screen for a while" (while refusing to explain why) to "it depends if your cloud is AWS" (not in any way remotely related to the question.) I did have a candidate follow-up with a series of questions about the drawing, which were feasibly legitimate.
This hand-written diagram is not an absolute filter (I've only used it maybe four times), but rather it can confirm some suspicions. I think I can generally gauge honesty from questions/tasks like this. And that's really what I'm after -- are you being honest with me?
It's imperfect, but it has been helpful.
Centralize from where the ts is set (the db is a great place to do this.) Don't let client-code set the timestamp.
> 2. Verify if it’s a problem from search volume.
It contextually depends, but correlating a problem-to-be-solved with search analytics can be really tenuous. I'd suggest a different phrasing:
Verify it it's a problem by speaking with customers.
Do it better the next time!
I haven't touched it in a while, but I've been thinking of a sync-to-another-device capability that I could then use with a locally-running AI instance to dig into. (Disclaimer: this is purely exploratory on my part; I'm sure someone will say 'why not do so-and-so'). As always, your mileage may vary. :-)
What I didn't expect was the amount of drama within the context of those papers. In Codd's original paper on relational theory in the 60s, he spends a chunk of time dedicated to talking about IBM as "the man". Hilarious.
We're not crazy about irregular cadences for project update communications. Why? Because there are many projects of complex size and shape going on. The stakeholders of my project care about many of those other projects as well, and they plan their schedules expecting to consume your update at that time.
Another case how we operate: the unpleasant surprise. Don't sugarcoat it, and bring things to attention as soon as possible. Rip the bandaid off, as we say.
As for tone and content, we focus on the purpose and delivery on that purpose. This is established early in the project, and our stakeholders understand both before the project ever takes off. We anchor our updates on those aspects, as it gives stakeholders an understanding of progress, no matter what the project update includes.
On one hand, this could easily be read as the musings of some overzealous, re-awakened and re-charged techbro/middle-manager who has tried and failed a few times. But this time, things are gonna be different....
But also, there's a modicum of truth in there. Only the experienced will recognize the red flags buried in the brush-away commentary.
The biggest complaint: "I don't understand why this takes so long."
Our stakeholders were mostly plant operations people -- users, not engineers. We re-framed the discussion to focus on WHAT they didn't understand. They didn't understand how software comes together, gets deployed or updated, etc.
I explained that it wasn't important that they understand how that occurs (or, if they do, please join the engineering team!) This literally took pressure off of them to be knowledgeable. I pulled a few of them in as "first-pass quality control", ensuring what we built would work for them and their teams.
Immediately, I had advocates for us and interested stakeholders willing to participate as partners.
On appearances, everything should have been just fine. The SWEs and product leader were running two-week sprints, with all the ceremonies scheduled on all calendars (well, everything except the end-of-sprint demo.) Every two weeks, a status powerpoint slide deck came from the product leader espousing all the fantastic capabilities that were recently implemented by the team.
I needed to understand what was going on. I chose to observe everything for two sprints (and not interrupt how everyone worked) while I focused on establishing a clear and comprehensive view of The Project.
What I discovered:
- An ambiguous and unclear definition of the project. Key facts, problem statement, overall solution ... did not exist. The product leader could give you some spin of this off the top of his head, which would invariably change from time to time.
- Work stories (we used JIRA) were garbage. I'm an engineer and even I couldn't understand what work was being completed.
- No capacity for the team to build & deploy software in a reasonable manner.
- No agreement or understanding of expectations among stakeholders.
So, what did we do?
1) Defined the project, with stakeholders in agreement.
2) Devised a solution, based on the project definition.
3) Broke down the work necessary to implement and deliver the solution.
4) Determined individual lengths of time to complete the list of work.
5) Reviewed everything, finding areas of risk and potential further problems.
It sounds like BUD (big upfront design) -- but in practice, it was Big Upfront Discovery. We structured work and components to ensure we had plenty of two-way doors (only a few one-way doors, based on requirements from stakeholders.)
And with that, we made our estimate. Another 6 months. The stakeholders did not like the timeline, but agreed "as long as we make that date." We provided weekly status updates, leaving out no details. Our stakeholder meetings went from contentious and confused to understanding and brothers-in-arms. We made the date and we didn't kill the team (morale actually went up.) The stakeholders loved the resulting solution.
****************
No magic here, just a focus on attention to details and honest assessment.
In my experience, homemade noodles will absorb sauce much better than the store-bought version. And, with a good sauce, the result is just _so much better_. You won't find me making my own noodles if they're going to be paired with Ragu, though.
Here's what is unsolved for me: store-bought garlic bread in foil to heat in the oven, or roll-my-own garlic butter and bread?
I recently completed a project on behalf of a privately-held company in the financial sector. Big, successful, and very corporate. The owner (a single individual) was in town and spoke to a few leaders (biz, tech, marketing) in my group. His message: get in the weeds, learn the business, understand every function, ask questions, get outside your comfort zone, be persistent, act with urgency, and finish things. (Those were literally my notes from that meeting.)
He gave his team room to do the things necessary to let their talent & smartness be most effective.
Also, Zuck: but hey everyone, keep on moving fast and breaking things!
Across the teams in my company, I hit nirvana when reviewing something like a major/impactful outage and the involved members in discussion are drilling down to the the essential state and sequence of events that caused said outage. Focus on actions and outcomes, including both those central/peripheral to the outage and those with zero knowledge prior to the event. It takes a high degree of trust in yourself, your peers, and your organization to get to nirvana.
Go through a few of these in a proper way, and a simple principle tends to emerge: if your mechanism is dependent on the perfection of humans, it will eventually fail. The only real discussion beyond that is basically what to do next -- do we need a mechanism that protects against human imperfections? Is the cost of implementing a solution worth the mechanism it would be designed to protect? Can we live with the infallibility of humans in this scenario?
Organizations that can achieve this level of discourse have a distinct advantage in execution.
One of the things I really enjoyed was seeing how differently all of the interviewees worked and went about their business. They didn't seem to work that much differently from me, which made their work that much more relatable.
- Format: we don't dictate format on docs. In terms of sharing important information, it's irrelevant. Dealer's choice.
- Audience: we are writing documents that are internal to our team, but also for consumption by other groups. Audience is engineering-focused.
- Consumption: we refer to documents as a way of learning all the things before asking uninformed questions to team members. RTFM is an expectation in my groups.
> instead of answering their very reasonable and pertinent questions you instead accuse them of being either incompetent or lazy.
These expectations aren't random; they are well-known, part of project deliverables, and non-negotiable. Existing projects show the quality of what's been produced in the past, to help with an understanding for anyone documenting what they've built and want us to depend on in production. After that, we definitely help those who still require help, but this isn't a surprise.
Nonetheless, I always have those team members who insist it's not their responsibility. A recent exchange I had with one of our engineers:
Eng: I can create our entire service portfolio, but I just don't know how to explain it in a document.
Me: I see. How about we start small, and just simply explain how service 'A' works. It's not too complicated.
Eng: Well, I don't know what outline to use. I don't know how to publish it. I don't know what content I should add. I don't know who would read it. This will take too long. It feels like busy work.
Me: So, if you're unable (seems like it) or unwilling (hopefully not) to do this work, what would you suggest we do to ensure we inform the rest of the company about service 'A'?
Eng: I don't know.
Me: Until we DO know, service 'A' will not be deployed.https://www.pinterest.com/pin/lol-code-geek-select-from-user...
(This shows the shirt you could buy once-upon-a-time from thinkgeek, which appears to be no more...)
I believe they used beefy-t brand shirts, which was like the nicest thing you could get for t-shirts at the time. (Personal opinion, don't @ me.)