News for hackers. Stuff that matters.
Re:What modern utilities should be a standard part of a mode... by crabbone on Saturday February 18, 2023 @07:06AM (#34846261)
As a long time user of utils-linux, I can say this: purge!

We need a big, big, huge purge of this ancient decrepit nonsense. We don't need blkid, lsblk, fdisk, wipefs, parted, df and a bunch of other lopsided semi-functional horrors to ask questions about storage. Just one program that works.

We need integration of MD, DM, LVM and possibly stuff like Btrfs volumes. ZFS was close to it. And commands associated with these utilities need to make sense in the way they are named and associated with each other. The way it needs to work is something like:

    $ storage volume-manager physical-volumes list
(This doesn't mean you have to type it out in full, useful abbreviations would still apply, but there must be a way to spell it out so that non-initiated may understand the purpose and the reason of the "spell")

Instead of guessing whether you should run pvdisplay or btrfs subvolume list

Networking seemed like it went in this direction sometime ago with iproute2, but it's still not enough. Stuff like nmcli or all kinds of nslookup, ss and so on need to be brought under one roof, generic enough to allow extensions and implementations by different concrete programs.

Today, just in order to find the configuration of one's network interfaces one must scout different places on their filesystem and try to figure out what utility is reading those and whether it's able to understand the config, and whether there's any other config in some other place that overrides the one they are looking at.

Re:What modern utilities should be a standard part of a mode... by eternityforest on Saturday February 18, 2023 @07:14AM (#34846325)
The more time goes on the more I start thinking a clean wipe of the entire thing might be in order. Linux is great and all, but it's so full of old crap.

It's getting better, but all the old crap still just sticks around fragmenting everything, and there's still no container system that can match APKs, because there's no API level to build to, every system is different, and everyone tries to support 50 different configurations.

Nmcli would be part of the OS itself if this were android or windows, and there wouldn't be ConMan and others unless you fork the whole OS or use some plugin API.

Re:What modern utilities should be a standard part of a mode... by sseagull on Saturday February 18, 2023 @07:31AM (#34846421)
But developing the new stuff doesn’t make the old stuff go away; there’s a huge userbase with trillions of lines of scripts and stuff to support.

It’s thinking like this that led to the fragmentation in the first place.

Re:What modern utilities should be a standard part of a mode... by dsr_ on Saturday February 18, 2023 @07:49AM (#34846518)
Right. A new project can:

- get changes into the kernel, and all newly installed systems after some point maybe three years from now will have them, probably

- get changes into the core requirements of a distro, and get newly installed and upgraded systems to have the feature in a couple of years, but only that distro. If the changes are too obnoxious, some organizations won't upgrade and some will leave for another distro

- make an implementation so attractive that lots of distros will include it either by default or as an option.

Nothing will get the project traction on systems that are not upgraded, regardless of the reason being can't or won't. And as long as people have a high expectation of working on those systems, the replacement will be another thing people have to remember.

Re:What modern utilities should be a standard part of a mode... by eternityforest on Sunday February 19, 2023 @03:57PM (#34861794)
Windows has lots of old systems, but they just continue to run the old systems for a few years, and then 20 more years after support ends.

It's a lot easier to maintain old versions when there are fewer of them and they are all corporate backed. One team can keep a version going. But with Linux there are thousands of versions, arguably millions because it encourages customization even beyond just choosing a distro.

With Windows, Android, and just about any other commercial style thing, there are versions that come out once per year or so, with either no forks, or forks that promptly get ignored and forgotten, or very minor customizations that don't really add or remove anything beyond the wallpaper and launcher.

And they generally preserve backwards compatibility better than Linux, because the stack can just be application+bundled libraries -> OS stuff fully controlled by OS vendor.

There's no systems made of small pieces meant to be interchanged by the installer, any one of which could stop working on a new system, all by different teams, you know your GUI toolkit works on the new OS version because it ships with the new OS version.

There are no other distros, because even when they are possible, like with AOSP, there's no culture of tinkering, people just take what they get, which both prevents fragmentation, and on the other side, gives the developers a reason not to make migration too hard, because they are trying to make one distro for all, not just a distro for their 10% of users.

Ubuntu does a good job of trying to be "The standard Linux", but things still change and break between versions far more than on Android.

Re:What modern utilities should be a standard part of a mode... by pdimitar on Saturday February 18, 2023 @07:51AM (#34846531)
Not necessarily, you can support the old stuff but require the new way going forward. Even better, you can script the migration of a lot of the old stuff to the new one.

It's pretty difficult to do but at one point we should get to it.

Re:What modern utilities should be a standard part of a mode... by eternityforest on Sunday February 19, 2023 @03:59PM (#34861808)
Android essentially does exactly that already with API level targets and it's great. They even have security features where they say "From now on, if you target this API level you can't do this without going through this API, and we won't allow new things in the play store below X target".
Re:What modern utilities should be a standard part of a mode... by crabbone on Saturday February 18, 2023 @01:06PM (#34849273)
Well, we have the administrative side of it down more or less. We have LTS releases, deprecation policies, even sometimes rudimentary automation to port old configuration and tools to match the new tooling. This part doesn't scare me.

The part that does scare me is that we'd need something like a task force that will have enough authority to compel all involved parties to follow their design as well as to have a good design in the first place.

Standardization is a powerful tool that enables significant advances in the industry that any individual player cannot accomplish on their own, and would be very beneficial to the most widely used operating system on the planet. But making useful standards is hard. Even harder in the presence of players who might want to achieve exclusivity for their product by obstructing standardization process. The later would've been my greatest fear.

Re:What modern utilities should be a standard part of a mode... by ilyt on Saturday February 18, 2023 @08:07AM (#34846641)
The problem is that it would need to be near perfect or else you're ending up with another kludged-in mess like networkmanager. I'll take my one-tool-does-one-thing-well over that anytime
Re:What modern utilities should be a standard part of a mode... by crabbone on Saturday February 18, 2023 @12:58PM (#34849185)
The problem is that currently Unix (well, Linux) doesn't have tools that do one thing right. They are too small and too overwhelmed by the problems they need to solve.

And this gets worse the more generic a tool you want to make, because, inevitably, you will have to account for the lower-level tools that your tool has to use. Let's, again, take storage as an example.

So, you want to write a program that shows users what block devices are currently available to them. Seems conceptually simple and potentially very useful. And, right off the bet you hit complexity: do loop devices count? Do separate partitions count, and if so, what to do about special partitions that keep data necessary for hardware RAIDs to function, that, in general, shouldn't be exposed to users)? How do you go about RAID devices, do you acknowledge that RAIDs exist and display them as separate entities, or do you pretend they don't and only display their constituents? How about physical and logical volumes? Do you think that you need to support ZFS volumes? What if some FUSE filesystem decides to expose some information as a device file, does that count as a block device? What about iSCSI (potentially with multipathing)? Encrypted devices?

And there are more... there's archaic NBD alongside with the blazing hot NVMe over IP and so on. And you don't really have any unifying interfaces. You can look into /sys/block, but depending on how you answered the questions above, you'll either find too little or too much info there. And there's not a single tool that can really, in all circumstances, give you a correct answer to the question "what block devices are currently available".

Unix started with a bunch of simplified assumptions and sloppy (if any) planning. It's OK for computers with predictable and not very diverse hardware configuration. But, because Linux is free (as in beer) enterprise sector decided to capitalize on this freedom. They no longer needed to develop the platform to run their stuff on. They took a tool that was supposed to enable individual users and small businesses to take ownership of their hardware and turned it into a money printing machine. And they keep using it w/o major investments as long as they can make profits.

It means that it sucks to be a system / infra guy (not financially, but in terms of what you have to work with), because your stuff is broken and will never get better, since the business decided that they'd rather run their stuff on a toy but free (as in beer) system than invest into developing a more grown-up one.

Re:What modern utilities should be a standard part of a mode... by ilyt on Saturday February 18, 2023 @06:34PM (#34852289)
> So, you want to write a program that shows users what block devices are currently available to them. Seems conceptually simple and potentially very useful. And, right off the bet you hit complexity: do loop devices count? Do separate partitions count, and if so, what to do about special partitions that keep data necessary for hardware RAIDs to function, that, in general, shouldn't be exposed to users)? How do you go about RAID devices, do you acknowledge that RAIDs exist and display them as separate entities, or do you pretend they don't and only display their constituents? How about physical and logical volumes? Do you think that you need to support ZFS volumes? What if some FUSE filesystem decides to expose some information as a device file, does that count as a block device? What about iSCSI (potentially with multipathing)? Encrypted devices?

I think PowerShell had right idea here, even if it maybe went too hard into "object oriented".

If tools had option to pass structured representation between eachother the above problem could be solved by tool that returns everything + set of generic tools to filter information, like say

    lsblk | f(type=disk,bus=nvme,name != nvme0n1) # all physical nvme devices except nvme0n1
    lsblk | f(type=lvm_lv,vg=data,mountpoint =~ {^/data} ) # any LVM LV that is in data VG and mounted under /data
I wonder if it would be possible to retrofit into existing design. Like, have additional filedescriptors and some way of signalling so shell can connect "structured out" with "structured in" of the next app in the pipe.

> And there are more... there's archaic NBD alongside with the blazing hot NVMe over IP and so on. And you don't really have any unifying interfaces. You can look into /sys/block, but depending on how you answered the questions above, you'll either find too little or too much info there. And there's not a single tool that can really, in all circumstances, give you a correct answer to the question "what block devices are currently available".

I'd be interested in which case lsblk is not showing block devices, so far it has been pretty bulletproof for me, even in extremely weird setups

Re:What modern utilities should be a standard part of a mode... by crabbone on Sunday February 19, 2023 @06:13AM (#34856564)
Microsoft isn't at the place yet where they may possibly have this problem. They aren't making an operating system that's extendable by others, that needs to incorporate interests of others. Where anyone may come and take any part of the system and repurpose it to their need. They have the luxury of not having to coordinate with others. This takes away from the usefulness of their product (eg. I see no value for myself in using such an operating system), but it makes it easier in engineering terms to make one.

So, PowerShell doesn't solve the problem. It didn't have one from the start. But, it also has nothing to offer to the Linux world as a consequence. The problem we have in the Linux world is more of a people problem before it's an engineering problem. We need an authoritative body that can mediate the interests of many parties using the system, incorporate their wishes and workflows, and then design a generic and extensible way of using the system. Right now it's chaos and anarchy which hinder or prevent the creation of useful programs. PowerShell would only increase the entropy of the system.

> I'd be interested in which case lsblk is not showing block devices

What about device files in a FUSE system randomly mounted somewhere? Also, I don't know of any way of showing loop devices with lsblk. Also, lsblk doesn't show RAIDs as individual devices, it shows them piece-wise. Also, I'm not sure about how it integrates (if at all) with ZFS block devices (I think it does on FreeBSD though). From what I understand, lsblk will only show a (large) subset of what may be found in sysfs. If it's not there, it's not a block device from its standpoint.

Re:What modern utilities should be a standard part of a mode... by LargoLasskhyfv on Saturday February 25, 2023 @12:27AM (#34934165)
Ahem. AIX: mk*, ch*, rm*, cp*, ln* almost anything.

The BSDs: ifconfig anything.

Any Linux? NIH everthing!