It is a thing, just not at that company.
Seriously, get a year or two, then look for escape options.
There's a Mt. Everest climb's worth of distance and effort between "it works" and "best practices", and like successfully climbing Everest, very few companies are doing all of the best practices, and I suspect your dude going the opposite direction by digging with a spoon.
Been a year and a half. Have a 2nd round interview lined up for Thursday to get off this crazy train.
I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;)
Since then the same patterns (alas, minus the diversity) have worked out a number of times and I’m adamant that adding another lead is far better than consolidating that under one person, at least until you have half a dozen leads.
In fact my most useful mentor and boss said to me look, I trust you guys to sort this out. I want to hear the plans and ask questions but I only need to be involved when you can’t come to an agreement. And then I can be your tie breaker.
I think a lot of things would go better if folks in these positions would audit and break ties and worry about bigger picture stuff most of the time.
I had a boss like this and very much believed in that sort of autonomy, only getting involved-as you said to be the 'tie breaker', but still lead a great engineering team and was a great source of knowledge and mentorship.
I say 'had' for a reason. That reason is mentioned in the post that started this comment chain. Now what's that tell you? Heh...heh :(
Given the politics there I suspect the rest of his time was kissing butts or covering them. Since the end of the dotcom era I’ve never seen a more high minded bunch of Emperors New Clothes types all in one spot. Woof.
Huh, I started reading that earlier today. Between the book and replaying levels in Infinifactory, the words "bottleneck" and "backpressure" loom large in my mind.
But then fast forward to the back end of your career and you've got a kind of weird dilemma. The vast majority of ridiculous things that the young guys try to push is absolute crap. They read about it in a blog post somewhere and don't have nearly enough experience to understand the nuances. Or they have yet another stupid f-ing framework that they just know is going to revolutionise the industry. Yeah... been there, done that, my closet is overflowing with T-shirts, thank you very much.
You sit there thinking, "I spent 20 or 30 years fighting my way through this crap and when I finally have a good handle on it, these new guys are just going to throw it all away". But... Dammit. You absolutely know that one of them has a great idea and you're not quite sure which one of them has it. Because they all think they have it sorted (even when none of them do). They feel they are completely sure that "this is the best way and only a moron couldn't see it". But you know that it's always more subtle than it appears. If you let them all run with their ideas, then 90% (99%???) will crash hard... Let's face it -- just like you did when you had that stupid idea that you were absolutely certain was "the way forward".
Anyway, I belabour the point. From my vantage point, it's pretty difficult to let go. Some people just can't do it. In my position now, I have a lot more sympathy for the "old guard" than I did when I was younger. To be honest, I think it is technically impossible for me to have less sympathy than I did -- and I think this is pretty common in the industry. Probably both sides of the equation are in a similar situation -- neither side feels comfortable letting the other side make decisions.
Just talking out of the hole in my head right now, but I think the main thing is that programmers generally like to program. They like to have freedom to try approaches that they think will be successful. As much as possible, it's probably best to allow your teammates, whoever they are, to have the freedom that they need. At the same time you should expect to have your own freedom. Getting the balance is not easy and sometimes you have to change jobs in order to get into a team where you are allowed the freedom to do your best. In my experience, teams that take this seriously are the best teams to work on (even if they sometimes do stupid things).
That said, in this specific context of tech (or at least in my wheelhouse of Ops), this is surmounted by creating and fostering a culture where it's okay to experiment and fail.
Where we've been handicapped in this regard is specifically not doing that, and the particular individual who makes this difficult having the mentality that any new approach spells doom and actively working against initiatives to put us in a better posture of experimentation in environments that don't sacrifice stability and reliability of our stack (hence the sarcastic comment about giving up my first born child for server immutability--I don't even have kids, haha).
But that said, I think you're absolutely right. I'm not quite greybeard yet-mostly because I yank out every grey hair I see trying to establish residency on my chin, but nonetheless your point is accurate, well taken and definitely that has immense merit.
And that means a huge cost of inconsistency. Every time you need to make a little tweak you have to learn how that part of the app works.
When you just want to get shit done, being able to go into any part of the app and knowing it's built exactly the same way as every other part means you don't have to spend any time learning insert X framework, you spend time doing, implementing business rules and fixing bugs.
A lot of programmers want to spend time playing with tech, not doing because playing with tech is more fun then doing the job.
Worst still, juniors (including me back then) are woefully unaware of just how clueless about programming they actually are, as alluded to by the parent, and that means they have massively over-inflated opinion of their views. They also tend to be wildly optimistic about how long a change will take and totally disregard the human cost (the cost of re-training and loss of productivity) of changing a piece of tech.
Meanwhile, you let him make decisions in his module/plugin/part/web service/whatever while you enforce rules between them.
Also, the work off critical path matters too, it would not be done at all otherwise, so treat it like it matters. It won't take juniors ego if older collegues take that job seriously.
As the technical field, we do not follow the scientific method.
Surely we should be looking to engineering here, not to science.
You clearly have no clue what was the predominant view in 1990s. Consistency wasn't just _a_ thing in the 1990s. It was a core dogma even beyond database systems. It was top of the agenda for a while until it sort of fizzled out in the early 2000s when distributed transactions hit a complexity wall.
For instance, you have to get all participants in a distributed transaction to support the same standard protocol without too much bridging, wrapping, special interface description languages, tons of configuration metadata, etc. You have to agree on a small stable set of features that everyone implements in a robust way.
But the major corporate sponsors of the competing approaches originating in the 1990s and early 2000s got disrupted before any one of them could gain enough traction to bludgeon everyone else into compliance.
My point was merely that almost no one in the 1990s doubted the necessity of consistent transactions. Eventual consistency and all the rest only became popular with NoSQL and the general desire to imitate internet giants that didn't even exist in the 1990s.
Don't let it happen to you.
I know that feel although in my case its non-technical management who just want things to be frozen in time.
Cest la vie at some companies unfortunately. Good luck with your escape plan. :)
Huh, that's exactly how I feel about the "must use a RDBMS for all data" senior architects I've worked with.
Eh, 'but we can't store things/API keys/passwords in VCS! It's bad!' is a pet peeve of mine. No, let's not 'just update it manually when we need it'. No, if outsider getting your DB password (somehow) poses existential threat... Chances are that you have way more concerning problems.
All it takes is one fuckup of accidentally committing to a public repo. Or firing someone on the team but not thinking to rotate secrets. Or hiring an outside contractor that now you implicitly give access to your production AWS credentials, etc.
Secrets do not belong in source control, hard stop.
If the secrets in source control are:
(1) encrypted
(2) never have decryption keys stored/loaded on any developer machine
(3) never have stored decrypted representations (only in memory representation of decrypted forms allowed when required).
If you follow these rules you won’t be more likely to accidentally commit unencrypted versions of the secrets and you’ll also by necessity have setup some (auditable) gatekeeper for logging of decryption events (via an aws kms or similar decryption as a service api).
For the category of secrets which must never exist in any decrypted form on a developer machine — maybe I can see the argument that those should be left out of source control as this would represent a reduction in the surface area for offline attacks against the encrypted form of the secrets -— but I would guess that this actually represents a somewhat minor gain in practice? And also seems strictly unimportant to use cases where access to the decrypted representation of the secrets on the developers machine is mandatory ...
Shameless plug, but check out EnvKey[1].
With 10-15 minutes of effort, it gets secrets out of git, out of email, out of Slack, etc. It manages encryption keys safely behind the scenes, protects development secrets just as strongly as production, keeps devs and servers automatically in sync, and greatly simplifies access control and key rotation.
It's not the only solution out there, but it's by far the easiest to setup and work with. In any case, use something that truly solves the problem! Don't settle for half-measures that end up spraying secrets all over third party accounts. This stuff is serious--even when it comes to so-called development secrets, the line is fuzzy.
1 - encryption keys and secrets are “hazardous material”, shortened to just “hazmat”. While necessary and arguably crucial in our work, they deserve the same care and respect a chemist would have for a beaker full of particularly dangerous chemicals.
If you have a team of 2-3 people, where everyone has access to everything anyway, and the secrets can't be used from outside the company (i.e. you need access to servers in the first place to authenticate with whatever service is configured), then it's one thing. It's bad, but it sure beats 'let's really carefully update it on all machines manually'.
I just really, really don't like blanket statements such as 'you shouldn't do that, full stop, context is irrelevant'.
Even then, secrets don't even belong in source control. They aren't tied to a specific version of your software -- they're tied to state in other systems: AWS, your MySQL database, a third-party API, etc. Those systems will change independently of any particular release of your code, and having it versioned in source control doesn't actually make sense.
You must track them somehow and a VCS is obviously the right thing to store them in.
Depending on the circumstances, I use git-encrypt, Ansible vault, vim -x (encrypt files) or even RCS (no chance of pushing to a repo when RCS doesn't have that feature).
This one of my criticisms of the 12 factor app. They say put the config in the environment. That doesn't address the question. How and why does it get into the environment and from where?
Storing secrets in version control (with permissions to the repo tightly controlled), encrypted (with the decryption key only available on the compute needing secret access), and used for rendering into a discovery mechanism (or directly injected into instance env vars) is entirely legitimate (and I know of several orgs doing this).
But otherwise yes, storing passwords in vcs is bad. That wasn't necessarily what I meant to imply by suggesting storing configuration files in vcs, however. I probably phrased that poorly.
Consistency in database systems refers to the requirement that any given database transaction must change affected data only in allowed ways. Any data written to the database must be valid according to all defined rules
I would LOVE for this to be a thing. Just as soon as I can convince people that maybe storing my.cnf in VCS is a better idea than updating it in prod. Live.
Sighs.