There were two stacks of these, six feet high, waiting for me.
It took me most of a year to work through the pile, finding out why the crash had occurred, by tracking pointers through memory with pencil and colored marker and comparing this with paper listings of the operating system. Then I'd code a fix for that problem, test it (usually around 2 AM when I could take over a mainframe), and nervously put it into production on one mainframe. Slowly, the piles of crash dumps got shorter and the mean time to failure went from hours to weeks.
There were very few meetings, and nobody interfered. They were just happy to see the crash dump pile shrink and the uptime increase.
After a few years of this, by which time the systems would stay up for months, I got a job in R&D at another company and got out of maintenance programming and into theory.
Yeah, it does. I’ve had a good amount of experience doing this. You need to have tact and a kind of humbleness that can be difficult to maintain (tbh, it was for me at least). In my most recent experience with this, many of the bugs were unknown. During my ramp up, I read through nearly all of the code I’d be working on and got a good sense of what needed to be done, but the code changes were the easy part. The issue that required more work IMO was navigating the social waters as a new hire.
I found that you can use being a new hire to your advantage, eg “hey I was reading through this piece of code and I had a question about XYZ.”
Sometimes I’ve felt certain bugs are, for one reason or another, difficult to discuss. One thing I’ve done in cases like this is to refactor a bit of code in a way that makes a subtle bug painfully obvious; then in review, where the reviewers were the original author and reviewer, it’s easier for them to see the bug and say “this looks weird,” to which I respond “ah, yeah, this seems like a bug, I’ll fix this.”
The rough part of this is that you can get so good at being the fixer that this becomes all you (ie. maybe you’re not working on new features, etc) which can suck. The bright side is that you can strengthen the team—-everyone gets a chance to build/rebuild context and mental models, design docs can be updated, and the teams foundation gets stronger as a result. Of course, ymmv.
As a new hire, I find it important to build trust and relationships.
My boss came to me and said "we want to move from AWS to GCP, using kubernetes and not have it be a mess like we have now"
I was new, senior, free from ownership. I chose to stay as a team of 1 so that I wasn't burdened with fires and meetings. It freed me up to put my head down and get the work done.
The grind was learning GCP, kubernetes, terraform, the platform/product, gathering requirements from several teams and mentoring people that would pop into the project as their time allowed.
I won't lie as learning new things was great. The project was partially greenfield as it was a platform migration and not a product rewrite. There were a lot of constraints & debt to deal with.
Maybe the new employee churn is one of the reasons that most startups get so much done... the new people are not burdened by ownership and being the subject matter experts. They are free to do things, good or bad. :)
Out of the frying pan and into the fire.
Total cloud spend was in 2 digit millions per year.
Many hundreds of k8s clusters, all using each providers k8s managed service.
My general high level take is this:
* AWS
Products mostly work quite well. Their UX is terrible and very inconsistent between products. But things mostly work, and their support is generally excellent, especially when you have a TAM (Technical Account Manager) who you can reach out to at any time. They will help you navigate any support issues, and ensure you get a quick resolution. But AWS stuff is generally very complex, overly complex in my opinion.
Also at scale API rate limits just kill you, and AWS continues to struggle to provide any real visibility into API rate limits. I'm told that large services internal to AWS have to work around this by building their own caching layer in front of any control plane APIs. Also the api limits are at the account boundary, so you have to start doing account sharding.
GCP:
Has better UX, more consistent experience. Much better visibility into limits and things like that. You still probably end up having to do project sharding at scale to work around limits. We continued to run into a lot of issues with GCPs load balancers blackholing traffic, and it was never really resolved. I think their support is generally worse than AWS. I've had substantially worse support experiences with GCP than AWS at 2 companies now. Mostly support not being particularly useful, and not resolving the issue. As a customer it feels like Google has struggled to adapt their culture to the needs of real enterprise customers. AWS has this much more figured out. But my GCP account teams have always been quite good.
Azure: The worst of all 3. Lots and lots of issues with their k8s service, low limits on the number of nodes you could deploy. k8s 1.18 supposedly fixed a lot. But their answer was always just that you have to upgrade. Their support engineers were frequently gave us actively bad advice that would have resulted in full customer outages if we'd followed it. The engineers on my team fortunately didn't listen and instead figured out how to fix issues themselves. Also frequently ran into various limits and you need to start thinking about subscription sharding.
Of the 3, I personally prefer using and interacting with GCP. But their support or is still certainly worse than AWS. That said, I always felt bad for our TAMs at AWS. It felt like a non-stop job where you had too be available at all hours to deal with unhappy customers. They usually gave us fantastic support, but I wonder at what personal cost.
Because the identity model is worse than IAM, and the quota system is insane. The global VPC networking being nicer doesn't make up for this, unfortunately.
I've tried to keep that level of intentional naïveté.
This is where a good manager makes a difference. They're also interested in your professional development. They should recognize the valuable work you're doing, have you train new(er) hires and move you into more senior positions, more interesting work, or both, whichever you prefer.
I agree with this statement and your other points. I’ve noticed a more insidious variant of this behavior: the expectation of interruptions. Some groups have such frequent priority shifts and/or a culture of fire fighting or door knocking such that even with a relatively open schedule, one is dissuaded from engaging in longer stretches of work for fear of having it all go to waste when the next meteor strike arrives.
We went from recognising that requirements may change after planning to zero planning and telling developers what's the next priority for the day, day by day.
This lack of planning and product definition is also what drives the lack of documentation, which is another big problem in 2021.
I don't expect going back to waterfall but also not what's going on nowadays. Hopefully we'll bounce back in the middle at some point.
The problem being. People don’t want to hear that it’s going to take them a while to get from 0 to 100%. They’d rather hear they can get from 0 to 99% in a day, then spend a month slogging through that last percent.
The wrong person in this type of role at a smaller company can things back significantly.
Now that I have more experience I realize he was more excited about cargo-culting big tech practices than about making sure our software was well-architected. We wasted so much time enthusiastically implementing his low-impact but technically exciting ideas.
A bad principal/staff person has around the same level of impact as a bad manager/director, except it tends to cause more technical badness than social badness.
[1] not the project board shape, but the original process it's named after where you set a hard cap on work in progress in each stage to find out where bottlenecks are.
As a person who’s worked in both american and european big companies... I can tell it’s not an american illness only and I have heard the same from asian friends. Best I got was one or two meeting-free days (at least on paper, in reality these would be ignored as well). I just got used to these things.
Ultimately, I've found the important bugs prioritize themselves no matter how sloppy your system is. The rest of the bugs are simply there to (1) provide context, reference (2) serve as fallback if product work tends to lag.
Close everything older than 2 weeks. If it’s important/relevant, it will crop back up. If it’s not, then it stays unfixed.
This can be really uncomfortable to do. But it’s how I’ve rescued a couple of teams that I’ve led as either an EM or Tech Lead.
Put another way - if everything is important or “must fix”, nothing is.
So to the author: that seemed like a waste of time. I would not have done it because I’m not convinced the outcome would have been meaningfully different from my tactic.
This context is really valuable when reasoning about the product or determining what's important to prioritize.
The issue with the "declare bankruptcy and important stuff will come back" is that you don't actually solve the underlying ruthless prioritization issue and very quickly end up in the exact same position.
You also irritate people that spent time writing up product issues for potentially important bugs by auto closing them (so many important things may not come back), but the bigger loss is that going through all of them gives you solid product historical context.
For what it's worth, the best PMs I know go through the backlog and understand what they're killing when they approach a project that's currently fucked. I think it has a positive side effect of giving them a lot of credibility too (as well as helping them ramp up).
Issues for those teams usually 'best effort' isolation going forward, which compounds the issue.
I mean, maybe not—you might lose the users impacted by the bug. Which from a company PoV is probably a loss, but depending on how broken incentives are may not be for the dev team.
I won’t argue that my way is the only correct way because I’m certain it’s not.
Something I didn’t make clear in the very brief description I gave was that, yes, capturing context of what was there is important.
That said - the two projects I worked in where we did this were filled with bad bugs (a separate and problematic issue).
Nothing really replaces spending time with users. That’s what I tend to opt for - figuring out what sucks in their experience and iterate on that.
It's great for individual contributors who don't care about understanding the product and are probably going to be in another team in 3 months.
In a previous job I did what the author did and it helped building an understanding of the project and with knowing at any time the list of the top X things we had to fix.
This seems like a great idea: technical debt bankruptcy.
In reality, you're throwing away hugely valuable data that people spent real time and effort producing for you.
See also: https://www.jwz.org/doc/cadt.html
https://hn.algolia.com/?dateRange=pastMonth&page=0&prefix=tr...
And of course that person didn't close their pet project or the features they promised.
It's slightly soothed when it's realized one of your summarily closed bugs could have prevented the next incident or major customer loss.
If I write a bug report, I'm doing the developers a favor by pointing out where they failed in development and validation. Simply closing the report with no action tells me that you don't care about the quality of your code or the way it affects the users; and that I've wasted my time thinking that you did care and writing up the bug.
About 6 months into a job I realized this was what I needed to do and totally cracked. Didn’t know the questions to ask or the way to learn what I needed to so I burnt out quick and quit. My fault for sure, but there was a ton of pressure to identify issues and fix them because senior people were “just too busy”.
Better to have meetings than to have what you’re doing cancelled several weeks in.
Bug reports aren't exactly sui generis, they don't just mysteriously appear.
It likely affects some people more than others depending on the demands placed on you.
Think I'm going request meetings never fall mid morning or mid afternoon.
Knowing about it doesn’t make it interrupt flow any less, unless you just avoid the kind of work that would require flow around it, which still imposes the same kind of productivity impact on intellectual “making” workers. [0]
[0] related: http://www.paulgraham.com/makersschedule.html
Especially since they don’t add any value, but that’s arguably a different problem.
Honestly, this is one of those traps a team can fall into, where nobody feels empowered to ignore the rest of the business for 3 weeks to put the bell on the cat. The only person without deliverables and due dates is the new hire. And it takes a special kind of new hire to have the expertise to parachute in, recognize that work needs to be done, and then do it with little supervision.
But he's right in general, that you can get some surprising things done by just putting in the time and focus. Which is why it's so utterly toxic that corporate America runs on an interrupt driven system, with meetings sprinkled carelessly across engineer calendars.