News for hackers. Stuff that matters.
Re:Systemd: The Good Parts by mjw1007 on Sunday May 16, 2021 @04:50PM (#27176913)
What particularly grated was that they managed to introduce bugs into udev and then say they didn't have the "expertise or understanding" to fix them: https://github.com/systemd/systemd/issues/7157#issuecomment-...

(That bug was causing udev to give multiple disks the same name in /dev/disk/by-path, which I think is pretty much core udev functionality, given that udev was intended to supersede devfs.)

Re:Systemd: The Good Parts by zxzax on Sunday May 16, 2021 @05:05PM (#27177007)
I'm just curious, how would you suggest they debug and maintain support for that hardware, without even owning that particular device? It seems like anything they do would just be a wild guess as to whether it actually works or not. In my experience with this specialized datacenter hardware, if people want any of this stuff fixed (in userspace or in the kernel) it usually falls on the hardware vendor to hire some developers to fix it and upstream it, or at the very least donate some hardware to the upstream project so they can test properly.
Re:Systemd: The Good Parts by mjw1007 on Sunday May 16, 2021 @05:19PM (#27177093)
If they felt they needed information from, say, the Linux scsi maintainers then they could have gone out and asked for it. If they thought that in order for udev to do a proper job with these devices they needed the sg3-utils maintainers to write something new, they could have gone and asked for that. As it was they just stuck a "needs-new-home" label on the bug and ignored it.

I think useful action would have been more likely if udev had still been maintained by a team who believed they'd taken on the responsibility to make /dev/disk/by-path work, rather than maintained as a small part of a project whose maintainers are mostly interested in other things.

(Also, IIRC from what I had to do to work around it, they could have fixed that bug by reverting the change that introduced it. It wasn't a case of existing udev code not working with new hardware; it was caused by someone making changes without understanding the consequences.)

Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @05:29PM (#27177166)
That's what I mean though, all of that seems to be just wild guessing as to what will actually work and sending random emails out to random people who may or may not be able to fix it (Disclaimer: I don't know anything about this specific bug or the specific hardware). Wouldn't someone who actually owns the device and knows who to contact on the kernel side be a better person to lead the effort on that?
Re:Systemd: The Good Parts by mjw1007 on Sunday May 16, 2021 @05:39PM (#27177235)
This was a bug in udev.

The right people to lead the effort on fixing bugs in udev are the udev maintainers.

If people did not wish to have the responsibility for leading the effort to fix bugs in udev, they had the option of not taking over the maintenance of udev.

I agree with ohazi, above, that the world would be a better place if the systemd people had availed themselves of that option.

(To be fair, it looks like this issue was fixed a few months later by someone who knew what they were doing. So the main lesson here may be something more like "@poettering should not be triaging udev bugs".)

Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @05:44PM (#27177280)
I see people saying this kind of thing often in open source and I think you are missing the point here: who is going to fix this bug? You can say systemd developers shouldn't triage it, and then you're left with nobody to triage the bug at all. So what will you do? The point is, they don't own the hardware, they can't fix the bug. You have to find somebody who actually does own the hardware who knows what is going on, which seems to be exactly what happened. But if you get unlucky and if zero of those people are contributing to udev or any of its forks, then the bug probably won't ever get fixed.
Re:Systemd: The Good Parts by mjw1007 on Sunday May 16, 2021 @06:08PM (#27177491)
For what it's worth, I didn't mean that the systemd people shouldn't triage the bug.

I meant that I suspect @poettering was mistaken when he said the current maintainers didn't collectively have "the expertise and understanding to maintain this properly."

As far as I can make out, there was nothing particularly hardware-specific about the bug. The bug was that udev was assuming that there could be no more than one disk per SATA host node, which turns out not to be the case.

`udevadm info` output would have been enough to see what was going on, which I'm sure the reporter would have happily supplied.

Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @06:17PM (#27177545)
From what you were saying earlier, from the perspective of the bug triager, it sounds like there were multiple areas that maybe this could have been fixed in, not necessarily udev. So still that comes back to: the correct people to triage that bug really would have been the distro maintainers, who have total visibility over the whole system and who are better equipped to pinpoint where the bug should be fixed. And then from there they can pass it off to a systemd person or an OEM person, or both, or whatever (Again disclaimer, I don't know anything about this particular SATA hardware or whether this is normal behavior for a host node, this is just my experience from trying to triage this type of bug).
Re:Systemd: The Good Parts by 3np on Sunday May 16, 2021 @06:06PM (#27177477)
I think their point is: with smaller project scope, this bug wouldn’t have existed in the first place.
Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @06:19PM (#27177556)
That is throwing the baby out with the bathwater though, people who want large scale projects will continue to make projects with large scopes.
Re:Systemd: The Good Parts by regularfry on Sunday May 16, 2021 @07:29PM (#27178242)
And they will continue to be bitten by problems specific to large-scope projects. Wanting something, no matter how much, does not let them escape engineering reality.
Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @07:42PM (#27178357)
I think they are aware of what they signed up for. Linux has continued to grow year after year, the reality for them is that they can afford it.
Re:Systemd: The Good Parts by regularfry on Sunday May 16, 2021 @07:52PM (#27178441)
Apparently not. If they had not signed up for it, in this case they would not have ended up triaging a bug they did not have the skills or hardware to tackle.
Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @07:55PM (#27178462)
Just to be clear: the alternative there is that nobody looks at the bug at all because nobody but the bug reporter has the hardware. I don't think that's what you want, I assume you would just prefer the bug to be fixed.
Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @03:15AM (#27180808)
No, the alternative is that, with no clear owner of the bug, the ownership problem gets dealt with early, rather than having the bug go stale in a can't-fix-won't-fix state for months because the wrong people claimed ownership of the subsystem.
Re:Systemd: The Good Parts by _d7dt on Monday May 17, 2021 @09:18AM (#27183050)
>the wrong people claimed ownership of the subsystem

That's... not what's happening at all? The issue is there _wasn't_ any of the right people around to claim ownership of the bug. It's not like someone in the know can't just look at the systemd bug tracker, it's all public. Like, I get what your complaint is, but at the end of the day, do you really care who's name is on the commit that fixes the bug? I usually don't, and most maintainers I know probably don't either -- they're usually happy to delegate to someone who's more knowledgeable in the problem area. If you have some other solution you'd like to suggest here then I'd love to hear it, let's move beyond the criticism and start thinking about solutions. And I don't even mean this as a solution in systemd, I mean this as a "helps anything that interacts with hardware and has bugs that could potentially be caused by hardware" solution.

Re:Systemd: The Good Parts by unanswered on Sunday May 16, 2021 @06:08PM (#27177488)
This wasn't a hardware bug and it didn't materialize only when some particular hardware was plugged in. Were they just supposed to open a phone book, pick a random hard drive manufacturer, and say "hey this is your problem now"?
Re:Systemd: The Good Parts by _d7dt on Sunday May 16, 2021 @06:21PM (#27177570)
No, it seems the specific hardware causing the issue was mentioned in the bug report.
Re:Systemd: The Good Parts by calvinmorrison on Sunday May 16, 2021 @10:14PM (#27179411)
systems/udev was spearheaded by employees getting paid to write it, not enthusiastic gnuers doing it for love.
Re:Systemd: The Good Parts by setheron on Sunday May 16, 2021 @10:41PM (#27179580)
My biggest gripe is that the logs are in a binary format.
Re:Systemd: The Good Parts by silly-silly on Monday May 17, 2021 @02:28AM (#27180581)
You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ?

# journalctl -u ssh

Re:Systemd: The Good Parts by rnhmjoj on Monday May 17, 2021 @03:02AM (#27180738)
I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work.

I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions.

[1]: https://github.com/systemd/systemd/issues/2460

[2]: https://unix.stackexchange.com/questions/86206

Re:Systemd: The Good Parts by zxzax on Monday May 17, 2021 @10:53AM (#27184130)
I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.
Re:Systemd: The Good Parts by throw0101a on Tuesday May 18, 2021 @08:37AM (#27193762)
> instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems.

Would SQLite be any more complicated than what they have now? Or OpenLDAP's Lightning Memory-Mapped Database (LMDB)? OpenLDAP/Symas' Howard Chu on the latter:

* https://www.linkedin.com/pulse/20140924071300-170035-why-you...

Re:Systemd: The Good Parts by lmm on Monday May 17, 2021 @03:08AM (#27180776)
Conventional logs are files, the fundamental unifying abstraction of Unix, and so you can read them with any tool, even a tool that was written before your logging system and that your logging system's author didn't know about. Having to take them just as piped input is a much more limited interface that doesn't allow you to do all the things you can do with a real file.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @04:41AM (#27181279)
> so you can read them with any tool

what tool can't you read them with? Why is the fs the fundamental unifying abstraction, as opposed to pipes?

> doesn't allow you to do all the things you can do with a real file

like what?

Re:Systemd: The Good Parts by lmm on Monday May 17, 2021 @08:13AM (#27182562)
You can't do random access through a pipe, so you can't binary search for it, or if you have some kind of probabilistic skip-sampling tool that won't work. Pipes don't have names, so anything that expects to work with them won't work. You can't re-read the same file. I don't know all the things you might want to do, but that's kind of my point - most things work with and expect files, and so if you want to be able to count on being able to use other unknown tools, a file is what you need.
Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @05:07AM (#27181434)
Pipes are files. From the point of view of the consuming process, you're just reading from a file descriptor. Files are more fundamental than pipes.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @05:29AM (#27181550)
If lmm shared this opinion, then there should be no problem, as systemd logs can be piped.
Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @05:35AM (#27181578)
All pipes are files. Not all files are pipes.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @05:41AM (#27181603)
So, back to the original question: what's the problem here? what is disallowed?

Also: If all pipes are files, but not all files are pipes, it would seems to me that files are more restrictive. That, and the extra steps you need to take to avoid needless IO.

Re:Systemd: The Good Parts by regularfry on Monday May 17, 2021 @08:06AM (#27182524)
I recommend https://www.amazon.co.uk/dp/013937681X/ref=cm_sw_r_cp_apa_gl... as an introduction to the fundamental concepts of the Unix filesystem. Certainly a better education than I could give here.
Re:Systemd: The Good Parts by Chris2048 on Monday May 17, 2021 @08:12AM (#27182553)
The original post was:

  that doesn't allow you to do all the things you can do with a real file
I'm not interested in what the differences between a pipe and a file are (I know) - I'm interested in why OP thinks they are relevant in this specific context/case; particularly in context of my statement:

  seems to me that files are more restrictive
Is, for example, a sticky bit relevant to systemd log files?
Re:Systemd: The Good Parts by kaba0 on Monday May 17, 2021 @03:16AM (#27180810)
But allows you to do things like properly filter based on multiple properties, and you can still use it to output text logs as well.
Re:Systemd: The Good Parts by lmm on Monday May 17, 2021 @04:11AM (#27181086)
There are plenty of unix tools for filtering files based on all manner of things. That's the unix way of doing things, and for many users the ability to mix and match unrelated tools is where a lot of the value of linux-like systems comes from.
Re:Systemd: The Good Parts by kaba0 on Monday May 17, 2021 @04:28AM (#27181207)
I don’t find grepping text logs for a service file name with many false positives better in any way than just filtering on an actual column of a quasi-db.

Also, you can pipe the output of journalctl to do whatever you want with it with those unix tools.

Re:Systemd: The Good Parts by deathcakes on Monday May 17, 2021 @04:24AM (#27181183)
Which I recall being touted as one of the major advantages of systemd - specifically that binary log files would make for much easier log aggregation as it would be more efficient to send over the wire. Fast forward to today and the best way of shipping logs is to force journald to output to, uh, text, then wring it through, yes thats right, syslog, to then be sent to a central server...
Re:Systemd: The Good Parts by TeMPOraL on Monday May 17, 2021 @05:35AM (#27181579)
At the risk of uttering a "you're holding it wrong" defense, it feels to me the issue isn't with logging format, but the stubbornness of *nix admins used to working with text. For better or worse, software moves in the direction of things large amounts of developers like the most, whether it makes sense or not.
Re:Systemd: The Good Parts by raffraffraff on Monday May 17, 2021 @02:27PM (#27186479)
Amarok 2 moved in the direction that the developers wanted, but the users hated it. Proves your point though!
Re:Systemd: The Good Parts by 867-5309 on Sunday May 16, 2021 @05:01PM (#27176985)
the systemd stars-to-issues ratio is worrying to say the least
Re:Systemd: The Good Parts by TheDong on Sunday May 16, 2021 @05:36PM (#27177221)
What do you imagine that ratio shows? Different projects use issues differently, so that number isn't super comparable...

And I've also noticed that low-level systems projects generally attract relatively few stars (i.e. a javascript library or go library will have more stars than a well used c library or systemd). My theory on that is that the people involved in projects like systemd, gnu, etc, see github as mostly just a git repository host, and don't bother with stars etc.

I guess from my perspective, I put zero weight into the stars on a project, and issue count only matters in the context of how the project deals with issues. What do you see as "worrying" in there? Do you see more value in those numbers than me?

Re:Systemd: The Good Parts by TeMPOraL on Monday May 17, 2021 @05:43AM (#27181611)
Github stars are just bookmarks. They're not an expression that someone loves the project, or are using it, or even very interested in it - they're just a way to bookmark something that piqued your interest, so that you can find it later.

This tells you why "low-level systems projects" attract less stars. It's because well-known, long-lasting projects attract less stars. There's a little point of starring a repo of a project that you recognize and use. You remember its name already. Meanwhile, a random JS library that looks like it could come in handy in the future? These things come and go, and you'll forget its name 5 minutes from now anyway, so you star it.

Re:Systemd: The Good Parts by saagarjha on Monday May 24, 2021 @08:22AM (#27263511)
Well, to you. I use GitHub stars as a way to mark projects that I use or particularly appreciate.
Re:Systemd: The Good Parts by 867-5309 on Sunday May 16, 2021 @10:40PM (#27179574)
I think it generally shows how active, responsive and sufficiently staffed the maintainers are. should projects be on github if the issues aren't used as intended? the worry comes from the number of systems in production relying on systemd
Re:Systemd: The Good Parts by TheDong on Monday May 17, 2021 @12:27AM (#27180045)
> should projects be on github if the issues aren't used as intended?

What do you mean by "not used as intended"? systemd has a mailing list too, but they also respond to github issues. Does having multiple different forms of interaction with your community mean you're not using issues as intended?

Some projects use github issues as a sorta backlog, or in conjunction with milestones and github projects to track future work. Some don't. Are projects that don't do that not using issues as intended?

As far as I know, github gives no general guidance on how to use github issues, and it's up to each project to setup issue templates and/or contributing.md files to explain that project's expectations.

> generally shows how active, responsive and sufficiently staffed the maintainers are

Okay, so if a project has a bunch of maintainers and is so well staffed they close all issues immediately, that means they have 0 issues, so that's good, right?

Except many well-maintained projects on github have hundreds to thousands of open issues, like nodejs with >1k, rust-lang with 7k, etc.

From where I sit, how well-maintained a project is has almost nothing to do with how many issues it has. Issues are a function of number of users (which is much different from level of maintenance), and maintainer policy on whether they close issues without enough information, close stale issues automatically, use issues to track backlog work, etc.

Re:Systemd: The Good Parts by detaro on Monday May 17, 2021 @12:38AM (#27180085)
So in your metric, all those projects that use a bot to close issues after 30 days as "stale" are using Github better?
Re:Systemd: The Good Parts by 867-5309 on Monday May 17, 2021 @11:48PM (#27191078)
it's not a metric, just an indicator
Re:Systemd: The Good Parts by kaba0 on Monday May 17, 2021 @03:20AM (#27180829)
There is for example nixpkgs, which uses issues very differently to a conventional repo, and you can’t really argue whether the project is over/under stuffed based on that, since it packages almost everything. Some of the issues may be upstream bugs, etc.
Re:Systemd: The Good Parts by pseudalopex on Sunday May 16, 2021 @11:02PM (#27179679)
More stars wouldn't decrease the number of issues though.
Re:Systemd: The Good Parts by beermonster on Monday May 17, 2021 @02:43AM (#27180649)
The criteria for a person to star a repo likely varies person to person. Therefore I’m not sure it’s a useful metric of anything in particular? Some crude popularity metric maybe? I don’t star repos myself. Feels more like a social networking feature.
Re:Systemd: The Good Parts by 867-5309 on Sunday May 16, 2021 @11:07PM (#27179697)
it would decrease the ratio
Re:Systemd: The Good Parts by pseudalopex on Monday May 17, 2021 @01:12AM (#27180257)
The number of issues has some relevance. The number of stars doesn't.
Re:Systemd: The Good Parts by 867-5309 on Monday May 17, 2021 @11:48PM (#27191080)
not following
Re:Systemd: The Good Parts by pseudalopex on Tuesday May 18, 2021 @02:16PM (#27198792)
You can't compare the number of stars for different kinds of software.

A large number of open issues means the maintainers can't keep up. 10x more issues mean the problem is worse even if the project has 20x more stars.

Re:Systemd: The Good Parts by TheDong on Tuesday May 18, 2021 @08:54PM (#27203315)
For what it's worth, I also don't think comparing issues between different projects works.

If you have project A, run by a company, which uses an internal Jira instance to track backlogs and developer's selected tasks, but uses github issues to track all user's reported bugs, well, that will have fewer issues than project B, which uses github issues to track backlogs and sprints in addition to bugs.

There's a lot of variability in this. The kubernetes project (~2k issues) has a bot that closes issues after 90 days by default (~9k closed by the bot so far), and without that bot, they'd have way more.

> A large number of open issues means the maintainers can't keep up.

That's not necessarily true. Github doesn't prescribe that every issue is immediately actionable. Perhaps the issue was tagged as "feature-request / help-wanted". Perhaps it is marked as "awaiting additional reporter info" (like logs, or OS), and there's nothing the maintainers can do until the reporter gets them more info.

Whether every issue gets triaged tells you something about whether maintainers are keeping up, but issues that are triaged aren't then always closed, so the number of open issues doesn't tell you much about how many issues are triaged and how quickly.

Re:Systemd: The Good Parts by Iolaum on Sunday May 16, 2021 @05:23PM (#27177124)
lolz - i 'll be keeping an eye on that from now on :) (on some of my fav projects)