News for hackers. Stuff that matters.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by xorcist on Sunday July 21, 2019 @02:17PM (#20492784)
> The one thing you never really do, as an ops person, is reach into the black-box components

I'm sorry, but just no. That's far too strong general statement. I've done quite a bit of sysadmin/devops style work over the years, and also worked with many other people who've sent various kinds of fixes upstream. Sure, third party software isn't always easy to fix, but it depends on what the alternatives are. You probably need to do an immediate workaround as well.

I would say that's an important part of why open source have had such a strong following among sysadmins, the ability to fix things. One could even say that diving into large pieces of software and exploring issues is the most rewarding part of the job. Just don't tell them I said that.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by zwischenzug on Sunday July 21, 2019 @02:23PM (#20492819)
Usually I've ventured into the code and made a bad enough attempt to fix that the devs are galvanised into doing it right. Filing a ticket usually not enough table stakes to get your voice heard if things are worked around.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by stallmanite on Sunday July 21, 2019 @05:12PM (#20493706)
I like this concept. I wonder if there’s a word for it. Any Germans want to help out with a fifteen syllable wonder?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by OJFord on Sunday July 21, 2019 @05:33PM (#20493803)
Tikettfileninsufficientstadtdelvencodebasefixen.

Or something? More seriously, relevant English idioms:

- A stitch in time saves nine;

- Something worth doing's worth doing well;

- If you want it done right, do it yourself;

and surely more. I'm certain there's a farming-analogy one about fixing something sooner rather than later, but it's escaped me.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by twic on Sunday July 21, 2019 @08:07PM (#20494515)
So, we've got:

Cunningham's Law: "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer" [1]

"Broke gets fixed, crappy is forever" [2]

I feel like there should be a third but that's what i've got.

[1] http://fed.wiki.org/journal.hapgood.net/cunninghams-law/fora...

[2] https://dandreamsofcoding.com/2013/05/06/broke-gets-fixed-cr...

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by Izkata on Monday July 22, 2019 @06:36PM (#20502929)
Maybe something about the dummy fix polluting the lead dev's code aesthetics?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by tgsovlerkhgsel on Sunday July 21, 2019 @07:02PM (#20494233)
Threat-in-a-patch
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by vast on Monday July 22, 2019 @02:55AM (#20495841)
Duct tape ops?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by nh2 on Sunday July 21, 2019 @04:34PM (#20493493)
Agreed.

I found that the best way to get reliable ops long-term is to mercilessly dive into the code for every issue encountered, and fix it at fundamental level.

It takes significant time, but after following this practice for a while, things start working reliably.

I'm currently doing this with software like GlusterFS, Ceph, Tinc, Consul, Stolon, nixpkgs, and a few others.

In other words, try to make it so that none of your components is a black box to you. Own the entire stack, and be able to fix it. That makes reliable systems.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by vageli on Sunday July 21, 2019 @05:41PM (#20493846)
> Agreed.

> I found that the best way to get reliable ops long-term is to mercilessly dive into the code for every issue encountered, and fix it at fundamental level.

> It takes significant time, but after following this practice for a while, things start working reliably.

> I'm currently doing this with software like GlusterFS, Ceph, Tinc, Consul, Stolon, nixpkgs, and a few others.

> In other words, try to make it so that none of your components is a black box to you. Own the entire stack, and be able to fix it. That makes reliable systems.

Do you then submit your fixes back to upstream? This is the critical element I think.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by caf on Monday July 22, 2019 @02:50AM (#20495825)
Do you then submit your fixes back to upstream? This is the critical element I think.

It makes sense to, otherwise you're stuck forward-porting your private patches forever. Or worse, get stranded on an old version because there's no-one around who can do that forward-porting anymore...