A few months from now, I'd love to have written down decisions for my current project:

- Why did I decided to use Kysely over Drizzle, Knex, Prisma, TypeORM or other ORM/SQL tool?

- How am I going to do migrations?

- Why am I using one of Deno/Bun over sticking to nodejs?

- Why did I structure the project as a directory per feature over controllers/models/services directories?

- Why did I fork this library and what are the steps to keep this thing updated? Do I plan to upstream my changes? Is there a GitHub issue or PR about it?

- Why am I hosting in one of AWS/GCP/Azure? Why not lambda functions? Why docker?

- Why did I pick this specific distribution of kubernetes over the other also lightweight alternatives?

- Why did I even start this project and what do I aim to accomplish with it?

So I created a # Decisions section in README.md

This way I don't keep doubting my own decisions and wasting time opening 20 documentation tabs to compare solutions yet again.

Technical decisions used to be in the daily-notes.org file, but keeping in a separate file makes it more accessible to LLMs. I actually started that practice before LLMs were in common use, I struggle to remember why.

Should that "why" be in technical-decisions.org or daily-notes.org?

Of course, you have a relatively high profile, so could probably avoid it/get it reversed.

I've also tried a mechanism where I have GitHub Actions write them out as JSON files in the repo itself, then I can git clone them in one go: https://gist.github.com/simonw/0f906759afd17af7ba39a0979027a... and https://github.com/simonw/fetch-github-issues

Lots of things can keep going with pen and paper or some cloud software.

At the very least, you have to communicate with your clients.

Um, sorry but what do you mean ?

Imo, nix is more finnicky but more of a complete solution than ansible.

Everything is quickly strapped together due to teams being understaffed. Preparing infrastructure in a way such that it can easily be recreated is easily twice the effort as “just” setting it up the usual way.

Either that bites you sooner or later, or you're lucky and grow; suddenly, you're a larger organisation, and there are way too many moving parts to start from scratch. So you do a half-hearted attempt of creating a backup strategy held together by duct-tape and hope, that kinda-sorta should work in the worst case, write some LLM-assisted documentation that nobody ever reads, and carry on. You're understaffed and overworked anyway, people are engaging in shadow IT, your actual responsibilities demand attention, so that's the best you can do.

And then you've grown even bigger, you're a reputable company now, and then the consultants and auditors and customers with certification requirements come in. So that's when you actually have to put in the work, and it's going to be a long, gruesome, exhausting, and expensive project. Given, of course, that nobody fucks up in the mean time.

I can go back to code I wrote months or years ago, and assuming I architectured and documented it idiomatically, I takes me only a bit of time to start being able to reason about it effectively.

With infrastructure is it a whole different story. Within weeks of not touching it (which happens if it just works) I start to have trouble retaining a good mental model of it. if I have to dig into it, I'll have to spend a lot of time getting re-acquainted with how it all fits together again.

I think this is an outdated view. In modern enterprises DR is often one of the most crucial (and difficult) steps in building the whole infra. You select what is crucial for you, you allocate the budget, you test it, and you plan the date of the next test.

However, I'd say it's very rare to do DR of everything. It's terribly expensive and problematic. You need to choose what's really important to you based on defined budgets.

Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server.

It turns out that the company that made this important piece of software doesn't even exist anymore.

This is why I like docker, if you keep the sources, you recover no matter what (at least until the "no matter what" holds water)

my understanding is that docker would not have helped in that scenario

Possibly with the same network settings if the licensing check was based on that.

But of course it can easily go south, though testing the recovery of a container based off an image and mounted volume is simple and quickly shows you if it works or not.

But of course it may work today but not tomorrow because the software was not ready for Y2K and according to it we are in the XX century or something and the license is 156 years ... young. Cannot allow this nonsense to proceed, call us at <defunct number>

IT is full of joy and happiness

yeah and that scenario was clear:

> Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server.

I am not sure I see your point, though.

This is around the time when you call that one guy on your team that can reverse engineer and patch out the license check.

This all got stored in the cloud, but also locally in our office, and also written onto a DVD-R, all automatically, all verified each time.

Our absolute worst case scenario would be less than an hour of downtime, less than an hour of data loss.

Similarly our dev environments were a watered down version of the live environment, and so if they were somehow lost, they could be restored in the same manner - and again, frequently tested, as any merge into the preprod branch would trigger a new dev environment to automatically spin up with that codebase.

It takes up-front engineering effort to get in place, but it ended up saving our bacon twice, and made our entire pipeline much easier and faster to manage.

"No, Restore is" I would say to stunned faces...

Sure it can help, but it's just not a one fix solution people thing. If you want a good test of your IAC, just provisioning a brand new environment first time using only your iac.

Huh? This is a strange assumption to make. Is your premise that IAC can't ever be truly reproducible?

If you are modifying things manually then you're not doing IAC.

The second rule of IAC club is that you do not hand modify your infrastructure.

How did they prevent threat actors presenting themselves as volunteers, were you vetted?

A far bigger risk is accepting incompetent volunteers if anything.

Half of the work is to know what you need, the other half is to know how you do it, while the third half is to cope with all the undocumented tinkering which happened along the way. So in that regard, starting from scratch can be acceptable, as long you are not starting from zero, and can build up on the knowledge and experience of the previous run(s). I mean, there is a whole gaming-genre about this, which is quite popular. And usually you have the benefit that you might be able to fix some fundamental failures which you had to ignore because nobody wanted to take the risk.

However, because in a previous life I'd been responsible for backups and involved in disaster recovery planning I was already kind of prepared with:

- a mirrored on site copy of backups (that they either didn't find or chose to leave behind)

- older hardware that had once been performing the duties of the existing seized gear (I'm a bit of a hoarder, I like repurposing or keeping for just such an occasion)

- multiple off site backups

- pretty good documentation of my setup

I was back up and running within a day or two and had lost maybe a couple of days of data. And it's a home lab, so nothing super important anyway, but a (not really) nice resilience test.

It also gave me the experience to work out a few structural changes to further limit the impact of an event that takes out a bunch of processing and storage.

(After 8 months they told me to pick up all my gear, they found nothing, but thanks for traumatising my kids)

Possibly the worst thing to be raided for: distribution of CSAM.

Apparently based purely on the 'evidence' of my IP address being on some list - that's the only explanation I ever got.

Funny thing is, they did so little background research they didn't even know to expect kids in the house when they raided at 6:30am.

It still triggers me. This was in August 2022. I wrote pages and pages of my memories and thoughts about it, and it still makes me angry for about ten different reasons.

The long version I haven't written yet and probably never will. I don't want to dwell on it, I want to get on with my life and have an even worse drama to deal with at the moment: https://news.ycombinator.com/item?id=44533637

I know I'm alive, that's for sure. I'm trying to make lemonade by the goddamn bucket load.

P.S. I have written prior HN comments referring to the raid if you care enough to go back that far.

I wasn't arrested or charged, they found nothing of what they were looking for on the multi terabytes of disks they seized. No further action other than the raid.

Sure you can sue anybody for anything. Whether your case actually gets heard or not is another consideration. And even if it gets heard, the judge can simply dismiss it for a variety of reasons before proceeding to trial.

Also, state and the federal governments have sovereign immunity and qualified immunity. Basically the government has to allow itself to be sued.

True this doesn't apply to counties or cities, however there is still a much higher bar for tort even for local police. Generally if they are operatikng within the law, like executing a valid search warrant, the standard is much higher than it would be for an average citizen.

Meerly suffering harm from government action is not sufficient. Having property impounded as part of an investigation, pursuant to a warrant, is likely not actionable, unless there was malice involved. Using slim evidence isn't really actionable.

We looked into anything that could be done to minimise the chances of such a thing happening to innocent parties, but the only option was to make a complaint about an individual officer. There's no (easy, obvious) way to question the system they use to determine "validity" of raids or due diligence prior to requesting a warrant, or evidence required to justify a warrant.

The whole thing just felt to me like it was blindly rubber stamped all the way through because "protect the children". Pity my daughter was a child and absorbs such experiences... My son was also a child, but he's less affected by such things.

And Japan, while being clean, safe and Kawai, its legal system has like a 90%+ conviction rate, so make of this what you will.

So, Norway, Dennmark, Sweden, Finland, Germany, Belgium, Netherlands, France, Spain, Austria, Switzerland

So what exactly are you referring to?

https://www.yahoo.com/news/german-woman-given-harsher-senten...

https://www.foxnews.com/media/germany-started-criminal-inves...

https://www.telegraph.co.uk/news/2023/08/10/autistic-teenage...

https://www.dw.com/en/germany-greens-habeck-presses-charges-...

Germany and the UK seem to be the worst offenders in regards to this, but that may just be selection bias on what gets into English language media. I can pull more articles if you're actually interested.

I think it's fair to demand that people follow laws on the internet as in person. Germany has laws against supporting nazism and for good reason. I don't give a crap about those people's right to free speech. The laws against insulting politicians seem a bit less reasonable but honestly just don't call people names online. Can't say I'm bothered by these articles.

Mention to them that the only way Communism and Nazism differ is Nazism following Engel's nationalist approach to overthrowing capitalism rather than Marx's international one and Nazis having been obsessed with spiritual and Iranian occultism and you're met with blank stares.

Yet those are the fundamentals of Nazism everything else derives from. Including the Holocaust.

Same goes for the understanding of the Nazis "Nationalism". Most Germans just won't believe you when you tell them that it was an absurd use of that term and much more similar to the modern "pro-EU" mindset than the modern understanding of nationalism.

The Germans are propagandized into thinking the definition of Nazis is being called "rightwing" by the media and wanting decentralization of power and cultural homogenity (modern nationalism). Yet the left:

- has already its SA precursors back on the streets (Hammerbande, Antifa, "NGOs")

- is getting rid of free speech to the maximum extent they somehow can pull off within the limits of the law and beyond that through mandatory voluntary industry initiatives (e.g. "Trusted Flaggers")

- are manufacturing external enemies at fault for all wrong (the Russians)

- is spinning up the war machine (massive military spending at 5% of GDP, near limitless commitment to finance the Ukraine war instead of trying to push for peace)

- Are centralizing more and more power in the hands of highly intransparent or even unelected institutions (e.g. EU or international contracts)

And various other stuff. E.g. some of the stuff you can hear on e.g. party congresses of the German "Die Linke".

But God forbid if you're ever caught for any crime whatsoever. Or if you're detained for domestic violence. Especially if not in Dubai (which is miles more lenient than other emirates).

When the police abuse their power, it's the community that pays their salaries that feels the pain.

My home was searched by the police for something much less serious (buying lab equipment, completely legally), and the experience left me having panic attacks every time there was a knock at the door.

Depriving people of their valuable property for 8 months or more is also abusively punitive. In warrants that grant seizures of all or most digital devices, judges should require police to return the items within 30 days if they don't either file charges or go back to the judge with good cause for an extension. If police can't get around to actually looking at the evidence they claimed was so crucial in 30 days maybe it's not a high-priority crime. And if having a reasonable time limit makes it too hard to look through so much stuff, they're free to more narrowly tailor their seizure requests so they don't have so much to troll through.

To possibly make this even more frustrating, when I was told I could pick up my gear, the detective in charge said that a few things they found flagged as suspicious:

1. I had / used virtual machines

2. I had "Tor" on my computer(s)

3. I had downloaded stuff from Megaupload

Now I'm not entirely sure whether these comments were based on what they found on my seized gear, or whether these were actually sufficient 'red flags' to make them think the warrant was justified initially, but, my god, how completely out of their depth, and therefore totally unqualified, they are to make life-changing adjudications about these things - and that their access to metadata only makes it more likely that they'll make false positive mistakes (which is just terrible for society overall).

I'm literally not sure what they meant by saying "you have tor on your computer", whether there's evidence of my having visited the dark web, or just having a (way outdated) copy of the tor browser saved somewhere.

And I think the only things I'd ever downloaded from Megaupload were Android ROMs.

Regarding Virtual Machines: I can't even... they're obviously non-technical so couldn't possibly understand, and yet... gah, I can't even...

My concern is around the sequence of events that needed to take place for this to happen to you. Also as a former network operator I want to know how laws like the data retention act, identify and disrupt, etc play a role in these situations - ie who triggered what. I think I’ll review your comment history.

Sounds like you have handled it in about as healthy manner as one could hope. I saw that as a compliment.

As someone who was arrested in his PJs at 4am due to a false accusation that the police did not investigate and for which they did not have probable cause, I feel this in my bones.

$15k in legal fees, a day in jail, and three months later, the charges were dropped because, as per the DA, “we cannot in good conscience pursue this case”.

No consequences for the person who made the false accusation, or the officer that enacted an arrest without probable cause.

My heart still skips a beat whenever I think I hear a knock at the door or noises in the middle of the night. I’ll wake up from a dead sleep in a panic. In theory I could pursue a lawsuit against both the accuser and officer, but that feels overwhelming — I’ve just tried to move on.

It completely changed the way I see the police and the criminal justice system. The process is, in of itself, punishment.

I was fortunate enough to be able to afford good legal representation, and I now have a great deal of empathy for those who are railroaded by the system because they cannot.

> It completely changed the way I see the police and the criminal justice system. The process is, in of itself, punishment.

One million times this. Even if it's not completely true at it's base, it's true for those unfortunate enough to have experienced this.

One of the funny (?) things about it, is that most of the individual officers were nice enough people, personable (other than one inflammatory officer who should not be employed in a role that comes into contact with people), one even gave the cat a pat. It's the system / systems, and likely the perverse incentives for their monthly stats.

I'm glad I didn't end up needing legal representation.

The way I deal with it is to actively let it go, forgive the situation in my mind, treat it as one of those anomalous circumstances that happen every second on a global scale. Treat it as a reminder of the unpredictability of life and an opportunity to be thankful that it's not my 'normal'.

Well,hopefully it has also changeed the way you see society in general, it's terrifying how common people willingly ( even when they have options) defer to the govt./authorities. The system didn't get be this this way overnight. Covid was one example of the tyranny of the majority.

I don't like that it can be interpreted that way, but I also refuse to stop using it since that's someone else's reading.

The only factor I use to treat one human different from another is whether they're a jerk or not, and I have to know them well enough to work that out. I've come across a few jerks in my time, and they take many and varied forms.

- daily backup locale + remote (blackbaze with 60 readonly retention strategy, separated bucket by service)

- monthly offline backup

- a preprod server where my users can restore entiere environment for testing purpose (CI)

in case of full house fire, i can be back online in an working day.

PS: i have only some TBs of data so quite easy to do.

most companies started rebooting the mainframe every six months to ensure they could restart it.

It went surprisingly well despite having stayed 15 years in the old DC without rebooting. They were super scared of exactly the case you described but except for some minor issues (and a lot of cussing) it was OK.

This is why I reboot my server from time to time after having applied patches or made more significant changes, despite the fact that "it should not change anything". This is a good moment to realize that it did change something and you have the opportunity to fix the issue while it I sfresh in your mind, and possibly with more time.

I'm curious about how you got in the door here. Very cool, but isn't healthcare IT notoriously cagey about access? I've had to do PHI training and background checks before getting into the system at my (admittedly only 2) PHI-centered jobs.

Granted, if it was such an emergency, I could see them rushing you through a lite version of the HR onboarding process. Did you have a connection in the hospital through whom you offered your services?

I volunteered to help because I knew that even broadly planning the recovery, evidence preservation etc. would be completely beyond the capabilities of the two IT folks (they were extremely nice and helpful, and glad that there was someone to help).

I was there to draw things on the board and ask the questions that will help to recover. I would not have (nor want, not have the need) to access patient information. This is something I warned them about early in the process, as the chaos was growing.

You need to imagine a large hospital completely blocked, with patients during an operation being stabilized and driven away.

I am used to crisis situations and having someone who will anticipate things you do not think about (how to communicate, how to reach prople having planned procedures, who does what and who talks with whom) is a useful person to have before the authorities kick in.

My wife had a planned operation that morning and I was on site when the ransomware hit, it is just this. Nothing James Bond like, just sheer luck to have been around.

The hospital made a recovery but it took about a year IIRC

Even the sequencing (recover and secure the network, then the AD, then some Tier-2 apps etc.) was something they were not ready for. I cannot blame them - the way these things are managed is really messy, with no clear responsibilities beyond the everyday operations.

My hope is that the continuous attacks on the national infrastructure (such as hospitals) will build a more coordinated and homogenous approach. This would be a great lesson learned.

The problem is environments like hospitals, who'll "cost center" their IT department to death, where even the most seasoned pro has no chance to ever do the right thing. There should be liability at the board level. There never will.

TBH your mistake was only running one layer of virtualization. What I do on my home setup is run a docker in a VM in a VM in a docker in a docker in a VM in a docker in a VM in a VM in a docker. This, I feel, ought to be the minimum level of indirection and virtualization in any technical configuration in perpetuity. Anything less is bush league and prone to errors.

What I meant I that I do not have the kind of setup a lot of people show: a DC-grade self, with a U2 switch etc. I have an old server, my ISP box, a UPS, a switch and maybe something else I forgot. But since the server runs both Home Assistant and Pi-hole, it is critically important.

OTOH this criticality allows me not to invest too much in monitoring: I have family and friends yelling immediately when something is down :)

Now it's simply become part of my engineering hygiene - as natural and effortless as brushing my teeth.

Actually drilling your DR is also crucial. If you never put it to the test, your documentation isn't worth the paper it's printed on.

In fact the last few years I've been thinking about ways for these systems to rebuild themselves on a continuous basis. Eg. I'd love a smartphone that competely restores itself from backup every night, even to brand new identical hardware, including secure element artifacts (either via private keys I securely control or reregistering everything in an automated fashion), with no user-noticeable impact.