News for hackers. Stuff that matters.
Re:Grub2 security update renders system unbootable by hinkley on Thursday July 30, 2020 @12:46PM (#24000291)
This is also why I pay someone to change my oil. It's a simple procedure, but the list of things that can go wrong is a list.

At a small shop I ran our VCS on my workstation (because otherwise we had none) and at the end of a day I rebooted to do some maintenance on the machine, and it didn't come back online until around 1pm the next day. On the plus side we finally got a dedicated server.

That said, you still gotta touch systems.

If you get off the upgrade treadmill, you will find yourself one day facing a CVE issue that have no patches for the versions you use. And then not only do you have to do a security update, you have to fast-track a version upgrade that your code might not work with anymore.

Last time we had this, I got the team to agree to putting one upgrade of OS or libraries per month onto the task list. We even let the assignee choose what they were going to upgrade (a few had no preferences, and so we'd have them work on whatever was most overdue).

We put some caches in place at work recently, to improve perf and give us some protection against partial outages. Already we are seeing services that cannot handle the same workload they could three months ago.

To paraphrase Jim Highsmith: there are no answers here, and we waste enormous amounts of time looking for them. There are only tradeoffs, with their own sets of consequences to be managed.

The tradeoff I prefer is to drill it into people that shit happens. Something is going to be down, and the more things we have, the higher the likelihood that one of them will be down at any given moment. You can put all of your eggs in one basket and have one outage a year, in which everyone not involved in fixing the damned thing might as well go home, or you have a bunch of systems where I can do 80% of my duties if any one of them is offline.

Systems that are never seen to break (or at least stumble) tend to become siloed, insular. Being able to talk about the 'bigger issue' around an outage is often a time when teams finally communicate with each other (unless you have dysfunctional managers, in which case: run)

And if all else fails, there is always some set of tasks you want people to do but they never seem to have the time for. Having a captive audience isn't the worst thing that could happen to you.

Re:Grub2 security update renders system unbootable by komali2 on Thursday July 30, 2020 @01:19PM (#24000683)
> This is also why I pay someone to change my oil. It's a simple procedure, but the list of things that can go wrong is a list.

This used to be me. I motorcycle, but unlike perhaps your typical motorcyclist I don't have a good brain/hand/mind for working on engines. I just tend to do a shitty job, get bored before I'm done, or can't wrap my head around the geometry of something (so I'll almost always turn a clutch adjuster the wrong way).

So I'd take my bike in for everything. But I was noticing that things didn't quite seem actually fixed, so I brought the bike over to MotoGuild, a place where you can rent time on a motorcycle lift and they have all kinds of tools you can use included, and popped the engine cover off.

I found horrifying evidence that I can't trust anybody to work on my bike other than me. The sprocket where the chain connects to the engine was loose to nearly falling off - it should be tightened to a shitload of newtons and secured with a locking washer. The clutch area was chunky with crud, despite me having paid for a clutch cable swap and them assuring me they had cleaned everything to avoid my new clutch cable getting gnarly and gross like the last one did (I'd lubricate it and black nasty would come out the cable cover). The chain wasn't really cleaned at all, they had clearly just run a rag on the outside, but between the links and on the tire side, it was nasty. Therefore, it wasn't actually effectively lubed, either.

Bunch of other random things as well.

I am like the LAST person to ever say "if you wanna get something done right, do it yourself," because I am flush with cash and bereft of discipline and willpower, but holy shit, apparently, if you wanna get something done right...

Re:Grub2 security update renders system unbootable by jacquesm on Thursday July 30, 2020 @03:02PM (#24001739)
On anything with two wheels you better be sure that you inspect it and if at all possible work on it yourself. Your life depends on just about every single bolt. Think ground bound helicopter levels of care.
Re:Grub2 security update renders system unbootable by cenal on Thursday July 30, 2020 @01:45PM (#24000989)
This is why mainframes running COBOL still are a thing.

If it works why muck it up?

People forget how dependent we have become on computerized automation.

Re:Grub2 security update renders system unbootable by linuxftw on Thursday July 30, 2020 @02:04PM (#24001205)
> This is also why I pay someone to change my oil. It's a simple procedure, but the list of things that can go wrong is a list.

If you're not going to change your own oil, take it to the dealership or a mechanic that you personally trust (I doubt this exists as mechanics don't really do routine oil changes). Putting the wrong amount of wrong type of engine oil into your engine is the absolute worst thing you can do to your engine.

I encourage everyone to change their own oil. If you run full synthetic (and you should), you will save a fair amount of money. Even if the shop gives you the right weight, and right amount of oil, you're not getting premium oil, and your oil filter is likely the equivalent of an empty tin can at most Tire and Lube type places.

OEM dealerships are the only places I trust to do any kind of fluid change if I'm not doing it myself.

Re:Grub2 security update renders system unbootable by boring_twenties on Thursday July 30, 2020 @05:20PM (#24002953)
> as mechanics don't really do routine oil changes

What's this now? Every single mechanic I've ever met is more than happy to do oil changes. Even my "BMW guy," who only works on BMWs and regularly works on $100k+ cars and fun projects like performance upgrades is more than happy to change the oil on my decidedly pedesterian BMW (12 year old entry-level 328xi, worth maybe $5-6k tops).

Re:Grub2 security update renders system unbootable by kevin_thibedeau on Thursday July 30, 2020 @01:10PM (#24000573)
I won't let anyone change my oil because a monkey once severely crossthreaded the drain plug by feeding it on with an impact wrench rather than hand starting it.
Re:Grub2 security update renders system unbootable by aidenn0 on Thursday July 30, 2020 @02:08PM (#24001245)
I'm fine with that because I just make damn sure they fix it, and if I have to fight at all for it, I take my business elsewhere.

I had a plumber crack the porcelain on the sink when tightening the drain. They told me about it and bought me a new sink. That's my plumber until I move. In another town the same thing happened and they put some silicone on it and didn't tell me. That plumber was never called again except to tell them why.

Re:Grub2 security update renders system unbootable by hinkley on Thursday July 30, 2020 @02:35PM (#24001472)
I failed an emissions test. The repair place near work wanted to replace the fuel injectors, it was going to cost a third of what the vehicle was worth.

The one near my house pointed out that there was chunk of EGR hose missing and the rest were ratty. They could replace them all for $280 (above the mininum required to address a failed test).

It was the hoses. Went from above the limit on two tests to 10% of the limit. I used that mechanic until I moved across town.

Re:Grub2 security update renders system unbootable by hinkley on Thursday July 30, 2020 @01:57PM (#24001123)
Oh, the pain is real. Cross threading is one of my nemeses. My eye still twitches when I see people cut a bolt on Youtube without threading the nut on first to work as a die (as in 'tap and die'). You're gonna cross-thread that thing! AAH.

I worked as a bicycle mechanic for a bit (lots of bolt cutting, more cross threading than I wanted), but also spent a cold winter working with my dad on a vintage car and I discovered that I don't have as much nostalgia for car work as I thought I might. For the same reason my boss pays for licenses (because you can yell at them and make them pay for the repairs) I pay others to do some of my handiwork.

Despite still being able to turn a bolt the right way while upside down and with the bolt facing away from me, I figure I'm almost as likely to get a rude surprise due to my own stupidity as theirs. If, and this is a big If, I've developed a relationship with the vendor, instead of just driving down to the Jiffy Lube where I'm nobody.

Re:Grub2 security update renders system unbootable by jacquesm on Thursday July 30, 2020 @03:01PM (#24001724)
Here's a simple trick that will help you to never ever have a cross threaded bolt: turn the bolt the wrong way until it clicks. That's the step of the threads disengaging from each other. Now they're perfectly aligned to mate so you move forward just a 10th of a turn or so and it will immediately engage. If it doesn't there is something wrong and you need to inspect the threads.