throwaway20371

My teammates create e-mail filters to send useless daily e-mail reports to the trash. I try to find out who controls the e-mail process, or the e-mail address, or something, so I can fix it... but I spend lots of time and it ends in vain. I couldn't find who "owns" the process, I couldn't find who controls the mailing list, and I couldn't get anyone to give me permission to change it even if I knew how.

Clearly the problem isn't just having the skill or permission to change something, it's also the friction involved in figuring out how the hell to do it. How do you lower friction? Documenting things, making it easy to find things, making it easy to get access to things. If you can come up with an internal system that combines all of that, you have a one-stop shop for fixing high-friction problems.

I think Wikis are highly underrated. They seem to encapsulate all those things. Anyone can edit (or revert edits), anyone can access it, anyone can find it (eventually). Somehow we need to tie all the rest of an organization into a Wiki.

So then shouldn't we stop writing software? If it's really impossible to make software that doesn't have tons of bugs, yet it's perfectly possible to make hardware without those bugs, shouldn't we be "making hardware" instead?

Actually, now that I think of it, that's not the problem. The problem is we keep changing the software. My laptop from 15 years ago still functions exactly the same way it used to. It hasn't disintegrated into a puddle of bits. You just can't use it to visit any "modern website" or run any "modern software". If we just stopped upgrading everything every 5 seconds we could keep using old technology.

It can work this way (that's how software patches have historically worked) but if you don't test it from the beginning, you will still find the odd case where that added step is broken, even though it seemed like it should have worked. The more you use that method, the more chances for breakage.

If you don't want to repeat the steps from the beginning, you could make a completely separate checklist to be followed on a given system that includes things like "make sure X package is installed", "make sure Y configuration is applied", so that the new checklist accounts for any inconsistencies. This is pretty common anyway as checklists are broken up into discrete purposes and mixed and matched.

That's basically what the do-nothing script is. The difference is that before you write any automation, you document all the steps in the script. Right there - when you've got it all written down, and no automation work has been done yet - that in itself is a very valuable piece of work. Now you can point anyone in the company to that script, and they can all accomplish the task without having to figure it out for themselves. You can now scale that process N times (N = the number of people in your company). Just writing down the steps has become a force-multiplier of repeatable work. Then as you begin automating each step, people automatically receive the benefit of that automation. Because the documented steps and the automation are in the exact same place, both will always be up-to-date.

What about the do-nothing script do you find is different than a checklist? To me the whole thing is already a checklist, there just aren't check-boxes.

> If you only install a webserver once in a blue moon, make a .txt checklist of the steps you followed.

This brings up a very important point about checklists that I don't think gets enough attention.

The problem happens when somebody "updates" that web server in-place. If they try to record what changes they made in the middle of the checklist, eventually when someone tries the whole checklist from the beginning, they'll find it's now broken; the steps aren't working as expected. This happens to me when I try to record changes in my VirtualBox configuration after I add a new system package or something; later I try to re-deploy my vbox, and it breaks.

So checklists should be considered immutable. Once you create them, don't assume they will work again if modified. Instead, if you make any change to the checklist, you must follow all the steps from beginning to end. This way you catch the unexpected problems and confirm the checklist still works for the next person.

In construction, if the foreman / lead whoever is always angry, people get fearful of speaking up about something, and then more mistakes get made because nobody wanted to point out the glaring flaw.

Soldiers also prefer trustworthiness over skill competency. Of course you want your brother-in-arms to do their job well, but it's more important that you can trust them with your life.

Technical skills are needed to work with a machine. People skills are needed to work with people.

You could call it an "E-I Script" for Efficiency Interest Script. Over time your costs are gradually lowered as each step is automated - like accruing interest in a savings account.

This is the way. I wish this were taught in computer science class, development bootcamps, operations team onboarding, anywhere there is a procedure that is even slightly complicated to automate. It is the absolute best solution there is.

* Documentation of the entire procedure is contained in one place. No need to go sifting through 20 different sources of documentation. This lowers the human emotional barrier to "just get it done", as people will always avoid things they aren't comfortable/familiar with, or don't have all the steps to. This central point of documentation also enables rapidly improving the process by letting people see all the steps in one place, which makes it easier to fix/collapse/remove steps.

* Automation in small pieces over time avoids the trap of "a project" where one or more engineers have to be dedicated to this one task for a long period of time. Most things shouldn't be automated unless there is demonstrably greater value in the cost of automating them than the cost of not doing so. Automating only the most valuable/costly pieces first gives immediate gains without sinking too much into the entire thing.

* One unified "method" to encapsulate any kind of process means your organization can ramp up on processes easier, reducing overall organizational cost.

* In the absence of any other similar process, you are guaranteed to save time and money.

I would say that the only potential downside is if someone decides to "engineer" this method, making it more and more and more complicated, until it loses its value. KISS is a requirement for it to be sustainable.

How is it not a fair comparison? They're machines. Just because we are currently building them in a way that is incredibly fragile and needs constant fixes, does not mean they have to be built that way.

Cars used to be built by hand, had tons of bugs, and were expensive. Then a man came along and found a way to produce them faster, cheaper, and with less bugs. That was pretty amazing for a time, but they still had plenty of bugs. And then some people from a culture of very fastidious craftsmen obsessed with quality began producing cars a little cheaper, and with far fewer bugs, and they lasted much longer. Then the whole world realized, "shit, our machines don't actually need to be so fragile," and they followed suit.

The lessons learned by those people in that culture were promoted around the world, and evolved to shape what we now call Lean and Agile. But the people using these new processes forgot the first lesson: we don't have to accept the status quo.

These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.

Why do your machines need monthly updates? Do you constantly update any other machine that you own? Lawnmower, car, oven, microwave, bicycle, watch, reciprocating saw, vaccum, garage door, TV?

Is Apple still the only company you can pay for both hardware and software support? Because all I want is to drop down a couple grand and never have to think about "computer maintenance" again. I maintain my car myself because it's so infrequent (pretty much just oil changes) but it feels like my computer maintenance is constant.

One of the reasons for that constant maintenance seems to be The Web. Remember when you didn't need 4 gigs of ram to browse the web? When you didn't need a high-power 3D graphics card to look at Google Maps? (bad example but WebGL is mandatory for some simple sites, and if your graphics sucks/doesn't do hardware acceleration...)

I don't remember ever having to upgrade my car every few years just to visit a new local business. At some point we need to admit that this constant tech churn isn't improving our lives, but it is enriching some billionaires.

I guess not if they're based in Estonia?

"MangoDB is a proxy which uses PostgreSQL as a backend. The proxy translates MongoDB wire protocol commands into SQL queries, and use PostgreSQL as storage."

You don't have to support MongoDB, but you can support apps that were only written with Mongo as backend? That's awesome. I can't imagine it's production-ready yet but it's a great idea.

It's not a philosophical difference, it's just complexity. More complex systems are more prone to failure. If the security system is more complex to set up, it's more likely to fail. More code means more bugs, and more domain-specific knowledge leads to more potential for user error. So if you have 'one program to secure it all', it's almost guaranteed to be better than having to use many programs all in the right way. And it isn't even a defense-in-depth issue because all those layers added to container security are really just to avoid the much larger attack surface; getting rid of attack surfaces reduces what you need to defend.

I've always used whereis instead of which anyway. More useful info out of one command.

Trying to secure a container via non-VM means is a painful slog. You can pretend containers give you security, and then one of the hundreds of different attack vectors provides a breakout. It's been demonstrated time and again, largely because Linux security is just shit and always has been.

I don't think there's ever been a year without a half dozen privesc holes in the Linux kernel. Linus is also belligerently anti-security because he thinks it always results in worse user outcomes. And containers were never created with security as a top priority, they're just an amalgamation of resource abstractions, so of course it works as well as anything else not designed with security in mind.

The hypervisor isolates guest kernel bugs from the host by nature of strictly controlling resource use from the lowest level. There are of course hypervisor bugs that allow breakouts, but they are a couple orders of magnitude rarer than the typical Linux privesc bug.

It's important to understand that containers are not a security device. Containers are a mechanism to separate resources used by processes. You should not assume any significant security benefits to containers, regardless of what anyone claims (even a kernel developer - maybe especially them....) because it all depends on Linux kernel security, which is pretty crap.

If you want security with containers, use Firecracker. It uses Micro VMs rather than just kernel-level restrictions, so even a Linux kernel security bug shouldn't be able to jump out to the host or other containers/Firecrackers.

> if { = 0 == 1 } then {

What in the heck is that first '=' for?

A job does not care about you. Always care for your own health and life interests above all business interests.

You should not quit immediately, and instead job-hunt so you can get an offer while still working. Hiring companies prefer to hire someone who is still working, and you don't want to burn your bridges (even a shitty company's employees are future bridges).

A lot of people in my company were smart and pivoted to working for FAANG from here, and I think they were here less than a year. In retrospect I wish I'd followed them.

> Sadly, when the 1% of the time hits this also means that nobody on the team has built up the DBA skills required for the big hairy problem. I see this pattern repeated all the time when consulting for startups that just had their first real traction and now the magic database box is no longer working as before.

I call this the Kubernetes effect. 99% of the time, people think it's easy. Then the 1% hits and they have to completely rebuild it from the ground up. For us it was cluster certificate expiration and etcd corruption, and some other problems we couldn't ever pin down.

The other effect (which I'm sure has a name) is when they depend on it for a long time, but then a problem starts to occur, and because they can't figure out a way to solve it, they move to a completely different tech ("We had problems with updating our shards so we moved to a different database") and encounter a whole new problem and repeat the cycle.

Nearly every single modern technology that exists, I have been trying to get away from for years. In fact, most technologies aside from the ones that maintain our houses (which are mostly only necessary because of the bizarre way we build houses) are detrimental to the environment and to our very health. Even health care technologies can be detrimental to us. We do live longer from the advancement of science, but technology itself seems to counteract that in many cases (not all obviously).

I don't want VR. I'm sure there is some reason why I should want VR. But I'm very happy at the moment not knowing what that reason is.

Luckily I have disconnected from social media, and only use things like YouTube to learn how to build a shed. But I dread the idea that some technology I don't even want will find its way into my life just because some company will make it impossible to live without it.

Good point - another data point that is actually not one, but many. There are 13 different COVID-19 vaccines around the globe, which work in different ways.

"The overall age-adjusted vaccine effectiveness against infection for all New York adults declined from 91.7% to 79.8%." - https://www.cdc.gov/mmwr/volumes/70/wr/mm7034e1.htm?s_cid=mm...

"Two doses of mRNA vaccines were 74.7% effective against infection among nursing home residents early in the vaccination program (March–May 2021). During June–July 2021, when B.1.617.2 (Delta) variant circulation predominated, effectiveness declined significantly to 53.1%." - https://www.cdc.gov/mmwr/volumes/70/wr/mm7034e3.htm?s_cid=mm...

"The efficacy figure, which is based on an unspecified number of people between June 20 and July 17, is down from an earlier estimate of 64% two weeks ago and conflicts with data out of the U.K. that found the shot was 88% effective against symptomatic disease caused by the variant." - https://www.cnbc.com/2021/07/23/delta-variant-pfizer-covid-v...

"“As seen in real world data released from the Israel Ministry of Health, vaccine efficacy in preventing both infection and symptomatic disease has declined six months post-vaccination, although efficacy in preventing serious illnesses remains high,” the companies said in a written statement. “These findings are consistent with an ongoing analysis from the Companies’ Phase 3 study,” they said. “That is why we have said, and we continue to believe that it is likely, based on the totality of the data we have to date, that a third dose may be needed within 6 to 12 months after full vaccination.” ...... Executives from Pfizer and BioNtech have repeatedly said people will likely need a booster shot, or third dose, within 12 months of getting fully vaccinated because they expect vaccine-induced immunity to wane over time. They also said it’s likely people will need to get additional shots each year." - https://www.cnbc.com/2021/07/08/pfizer-says-it-is-developing...

So, while it's true that the effectiveness wanes, the same articles also point out that it is still effective at preventing severe hospitalization. If you're vaccinated, you're more likely after 6 months to get COVID anyway, but you are much less likely to be sent to the hospital or die.

> It’s a respiratory virus. Not unlike many of its kind.

It's very much unlike others of its kind. And it is novel. It is literally called a "novel coronavirus". Just to give you an idea how deadly and different this virus is:

- An outbreak of Ebola from 2014 to 2016 killed 11,310 people in West Africa.

- In 2009, the H1N1 pandemic killed approximately 12,469 people in the US.

- In 2014, during the MERS-CoV pandemic, 2 people in the US tested positive.

- As of today, COVID-19 has killed 661,000 people in the US.

COVID-19 is 53 times more deadly than the last major coronavirus pandemic in the US. Fifty. Three. Times.

> We have a vaccine for it.

We have a vaccine developed for the first variant of it, alpha. The vaccine was not developed against the delta variant. There is a drastic difference in its effectiveness against delta. Hence why there are now new mask mandates - for vaccinated people - where delta is rampaging. Are you telling me you didn't know this at all?

> I’ve done my part. My obligations to society are over.

Actually, no, society literally requires you by law to continue to wear masks in places where Delta is surging. But whatever; you want to kill people, you're gonna kill people.

I get it. You're petrified. But life will get better, and it does indeed change. "Normal" does shift. People didn't used to wear condoms, now they do (except for you). People didn't used to brush their teeth, now they do (except for you). People didn't used to wash their hands, now they do (except for you). You apparently live in 1750, where medical science and overall culture hasn't changed for hundreds of years, where everything is totally normal and will never ever ever ever Ever EVER change. Because everyone else is just insane, and you're normal.

Denial is a nicer place to live than reality.

I quit most social media (I'm still here, aren't I?) and I found a lot more peace. I ended up working a lot more, which stressed me out, but I finally identified my lack of work-life balance, and now I'm back to a healthy state. No outrage at irrelevant trivia, no other people's lives being a distraction in mine. If anything I just have too much personal stuff to do now. Life feels a bit more meaningful, healthy. I recommend it.

Well that's because it's not a development workflow. That's a live systems change. It's like changing the tire on a moving vehicle.

If you have a very repeatable environment, you can have an entire pipeline that creates new infra from scratch (w/Terraform), build and deploy your new app, test it, and then point traffic at the new infra. It's like blue/green but bigger. You aren't changing the tire, you're moving from one moving vehicle to another one. That works well because there's no chance for unusual problems from trying to figure out how to re-jigger things on the fly.

The former is configuration-management-organized infrastructure, and the latter is immutable infrastructure.

The problem comes in with things like changing an S3 bucket or IAM role. Changing those things is like changing the highway... you can't replace the highway. You have to close down a lane of traffic, put up traffic cones, reduce the speed limit, make your changes carefully. Ideally test on a strip of test highway first.

These cloud-managed services cannot be made immutable, so you have to use configuration-management. So you have to have a change management system in place, and tightly manage the dependency between your app and the change.

"Show: open a pull request for review, but merge into mainline immediately"

This is literally the opposite of Shift Left. Merging in crap and talking about fixing it later. This should be renamed the "Tech debt accelerator".

I'm a simple man. I see article says "We don't know what this ancient word means", I upvote.