News for hackers. Stuff that matters.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by derefr on Sunday July 21, 2019 @01:31PM (#20492473)
> It all feels like a situation where the people maintaining this code knew deep down that it wasn't really up to scratch, but were sufficiently used to workarounds ("use direct io", "scrape dmesg") that they no longer thought of it as a problem.

It's an ops-level solution. The one thing you never really do, as an ops person, is reach into the black-box components that make up your infrastructure to fix them at an architectural level. That's not your job. The system is already in production; it's your job to make it run, without changing it—to shim around the black boxes to make them work despite their architectural flaws.

Sure, you can file a ticket with the dev team about how dumb you think an architectural choice was—but, in most businesses, most of the black-box components you're working with are third-party, so filing a ticket isn't likely to get you much. (Google is an exception, but even Google has to hire their SREs from somewhere, and they'll come in enculturated to the "it's all black boxes and we're here to ziptie them together" mindset.)

And, even in an environment like Google's where you have access to the dev-teams of every component you're running, dev cycles are still longer than ops SLA deadlines. Ops solutions are chosen because they're quick to get into production. (Component can't handle the load because it's coded poorly? Replicate it and throw a load balancer in front! Five minute job.) So even if you do file that ticket, you've still got to solve the problem in the here-and-now. And once you do solve the immediate problem, it's no longer a hair-on-fire problem, so that ticket isn't going to be very high-priority to fix.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by xorcist on Sunday July 21, 2019 @02:17PM (#20492784)
> The one thing you never really do, as an ops person, is reach into the black-box components

I'm sorry, but just no. That's far too strong general statement. I've done quite a bit of sysadmin/devops style work over the years, and also worked with many other people who've sent various kinds of fixes upstream. Sure, third party software isn't always easy to fix, but it depends on what the alternatives are. You probably need to do an immediate workaround as well.

I would say that's an important part of why open source have had such a strong following among sysadmins, the ability to fix things. One could even say that diving into large pieces of software and exploring issues is the most rewarding part of the job. Just don't tell them I said that.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by zwischenzug on Sunday July 21, 2019 @02:23PM (#20492819)
Usually I've ventured into the code and made a bad enough attempt to fix that the devs are galvanised into doing it right. Filing a ticket usually not enough table stakes to get your voice heard if things are worked around.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by stallmanite on Sunday July 21, 2019 @05:12PM (#20493706)
I like this concept. I wonder if there’s a word for it. Any Germans want to help out with a fifteen syllable wonder?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by OJFord on Sunday July 21, 2019 @05:33PM (#20493803)
Tikettfileninsufficientstadtdelvencodebasefixen.

Or something? More seriously, relevant English idioms:

- A stitch in time saves nine;

- Something worth doing's worth doing well;

- If you want it done right, do it yourself;

and surely more. I'm certain there's a farming-analogy one about fixing something sooner rather than later, but it's escaped me.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by twic on Sunday July 21, 2019 @08:07PM (#20494515)
So, we've got:

Cunningham's Law: "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer" [1]

"Broke gets fixed, crappy is forever" [2]

I feel like there should be a third but that's what i've got.

[1] http://fed.wiki.org/journal.hapgood.net/cunninghams-law/fora...

[2] https://dandreamsofcoding.com/2013/05/06/broke-gets-fixed-cr...

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by Izkata on Monday July 22, 2019 @06:36PM (#20502929)
Maybe something about the dummy fix polluting the lead dev's code aesthetics?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by tgsovlerkhgsel on Sunday July 21, 2019 @07:02PM (#20494233)
Threat-in-a-patch
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by vast on Monday July 22, 2019 @02:55AM (#20495841)
Duct tape ops?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by nh2 on Sunday July 21, 2019 @04:34PM (#20493493)
Agreed.

I found that the best way to get reliable ops long-term is to mercilessly dive into the code for every issue encountered, and fix it at fundamental level.

It takes significant time, but after following this practice for a while, things start working reliably.

I'm currently doing this with software like GlusterFS, Ceph, Tinc, Consul, Stolon, nixpkgs, and a few others.

In other words, try to make it so that none of your components is a black box to you. Own the entire stack, and be able to fix it. That makes reliable systems.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by vageli on Sunday July 21, 2019 @05:41PM (#20493846)
> Agreed.

> I found that the best way to get reliable ops long-term is to mercilessly dive into the code for every issue encountered, and fix it at fundamental level.

> It takes significant time, but after following this practice for a while, things start working reliably.

> I'm currently doing this with software like GlusterFS, Ceph, Tinc, Consul, Stolon, nixpkgs, and a few others.

> In other words, try to make it so that none of your components is a black box to you. Own the entire stack, and be able to fix it. That makes reliable systems.

Do you then submit your fixes back to upstream? This is the critical element I think.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by caf on Monday July 22, 2019 @02:50AM (#20495825)
Do you then submit your fixes back to upstream? This is the critical element I think.

It makes sense to, otherwise you're stuck forward-porting your private patches forever. Or worse, get stranded on an old version because there's no-one around who can do that forward-porting anymore...

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by mjw1007 on Sunday July 21, 2019 @01:38PM (#20492516)
I don't think there was anything wrong with Google using this workaround.

But Google were also employing the ext4 maintainer, and fixing those components presumably was part of his job.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by kbenson on Sunday July 21, 2019 @01:49PM (#20492589)
It depends quite a bit on the company and the software they are running. I've worked at places that use open source for the majority of services they run on the servers specifically because they can both look at the source as an advanced form of debugging, and provide local patches to software until it's accepted upstream (or indefinitely if it's not accepted or too company specific).

A lot of patches for open source projects come from companies like this. Is there also a lot of working around buggy software? Yes. It's not always that way though, and if it's gotten to the point that you have devs for that software on staff, it really shouldn't be, because if those people who are financially motivated to care don't, who will?

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by derefr on Sunday July 21, 2019 @02:06PM (#20492707)
> I've worked at places that use open source for the majority of services they run on the servers specifically because they can both look at the source as an advanced form of debugging, and provide local patches to software until it's accepted upstream (or indefinitely if it's not accepted or too company specific).

Sure, but what you're talking about here is more "the developers of your software, solving problems in the composed release of the software, by vendoring + patching infrastructural dependencies."

Like, for example: my software depends on Postgres. So I have a copy of Postgres running on my workstation, to develop against. It's built from source. If I find a bug in Postgres, I can fix it in the source, submit a PR, and in the mean-time, package this fork of Postgres into the application's Docker image in place of the mainline one.

This is basically the same as saying "my project depends on ecosystem-package libfoo. I can vendor a copy of libfoo to fix a problem with libfoo."

Either way, that's still something only the developer of the software can really do (on any kind of a practical time-scale.) Changing the software from talking to an external Postgres, to talking to one packaged inside the Docker image? That's an internal architectural change that requires rewriting the software's configuration handling logic and build scripts at the very least, and probably also adding some sort of init daemon to the resulting Docker image. It's a software change. It's not something an ops person should be spending their time getting up-to-speed enough on a project to be able to do. It's the thing they file a ticket to ask the developer to do.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by jayd16 on Sunday July 21, 2019 @02:07PM (#20492716)
As a dev, I hate this sentiment of siloed ops.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by derefr on Sunday July 21, 2019 @02:13PM (#20492758)
It's kind of impossible to have it any other way. Ops is a cost center; you'll only have as many ops staff as you need to ensure your hair isn't on fire. And so ops staff only have enough time to solve problems with the most direct shortcut (usually by adding more components into the mix), rather than doing good Engineering to pay down the technical debt responsible for the problem.

If you have an infinite budget, by all means, hire developers whose jobs are to be prophylactic for ops tickets, by paying down tech-debt. But you'll have to guard that department with your life, because it'll be the first thing to be cut the moment anyone's looking to grow margin by shaving costs.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by dgoldstein0 on Sunday July 21, 2019 @06:17PM (#20494021)
That is, until your customers start demanding higher SLAs and offering money for them... Maybe only a thing for really big companies, but when that happens, improving reliability and ops becomes someone's job. If not multiple people.

That doesn't mean no duck tape like solutions get used, but that the ones that cause problems get removed.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by KaiserPro on Sunday July 21, 2019 @03:24PM (#20493157)
I don't think thats siloed.

A good File system maintainer is a precious resource. wasting them on running stuff is madness. (I say that as a Devop.) I know a fair amount about running/configuring/tinkering with filesystems. (hell I've bumped into at least 3 XFS bugs, ones that were new to Redhat.) But I would never presume to be able to make a meaningful patch.

Its the same way that I'd not expect the Ext4 maintainer to know how to instrument, metric and monitor a large company's infrastructure.

Almost certainly they would have flung a ticket over the fence, and it was up to politics to make it work.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by jayd16 on Sunday July 21, 2019 @03:54PM (#20493303)
Its the fence that I dislike. But I work in a small studio where I can easily work with (and do some) devops.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by techslave on Sunday July 21, 2019 @06:45PM (#20494145)
i have some bad news for you. that’s not how ops at google works. root cause solutions are preferred and rabbit hole explorations are encouraged.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by jimmaswell on Sunday July 21, 2019 @01:33PM (#20492484)
If fixing the thing itself is easier, and also benefits everyone else, why not?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by derefr on Sunday July 21, 2019 @01:40PM (#20492531)
Fixing the thing isn't usually easier. In an ops environment, you're not dealing with Git repos full of source code that you can just patch and run through CI; you're usually dealing with pre-built packages from an apt server, or pre-built Docker images, or pre-built virtual-appliance VMs. And usually these are all created in drastically-varying, heterogeneous ways, many again by third-parties (public Apt packages, public Docker images, etc.), but even if internal, then by different internal teams with different favored approaches toward building and deploying things.

From the position in the pipeline where ops people operate, it's more work to figure out how to fork a build of software X, patch it, build it, and get out the same kind of artifact your deploy pipeline is expecting, hosted in the same place; as it is to just throw more spaghetti at the wall to solve the problem.

Half of the reason behind Google's monorepo, and the focus on Golang, is to empower SREs to patch broken components when possible, by making there be only One True Way to do that, and only one language (without weird per-project DSLs) to know. But that still doesn't completely fix the problem, since you still need to understand the architecture of the software project, and sometimes that's the hardest part.