You can turn off swap if you don't need hibernate, and from memory even turning off overcommit used to be OK-ish (of course most software written for linux doesn't try to deal with failing malloc requests gracefully, because there's no point since it never happens in the default configuration). You end up with a noticeably snappier system. However, this no longer seems to work in practice. Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory.
> A far better approach is to have your indexer run as a separate process that monitors file writes (you can still have a kernel hook for that if you wish)
This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration). I agree that you don't want to synchronously update all indexing meta info on FS operations because everything will grind to a halt if you do that. But you still want OS support such as reliable notification and extended FS attributes to store things like "this was downloaded from here" or tagging. I don't think there is anything particularly magical about this (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages) but in practice xattrs end up pretty much useless in linux because next to nothing uses them (baloo probably does) and as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though.
I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days.
Do people still hibernate? I thought these days suspending was a solved problem.
> Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory.
I thought the point of this discussion was talking about sane defaults? Of course if you're going to mess with kernel parameters then you run the risk of getting undesired behaviour. It's no different to when we used to tweak the BIOS in the 90s. So I'm not going to disagree with you there. But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts?
> This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration).
That's not file system integration though. What you were actually describing was a completely different behaviour. Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue.
> (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages)
Again, you don't want that information in the file system table. Storing every little bit of information like that in xattr would slow down standard file system operations. What you actually want to do is store that information in a separate RDBMS (eg sqlite3, MySQL/MariaDB, etc). To be honest even something like Redis might work as long as it has a persistent backup.
> as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though.
I've not spent a great amount of time with inotify but from my limited exposure I do recall it wasn't great with nested hierarchies. There's probably some better ways that I don't know of but this is a particular problem I've not needed to solve before so I'm as in the dark as you are.
> I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days.
Honestly, I think the perceived differences are all imaginary. Like wine tasting when you're told one bottle is expensive and another is moderately priced - lots of people will start to imagine deeper flavours in the more expensive bottle even if those flavours don't exist. So much of our perceptions are based on expectations rather than experiences and I think that's what's happening here because I've used both Krunner and Spotlight and my honest impression is that they're both much the same.
I should probably have phrased this differently, "kernel file system layer integration" maybe. The relevant (and presently, I believe, lacking part) in linux would be VFS. It also relies on applications making consistent use of xattrs for some functionality, something that does also not happen on linux.
> Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue. [...] Honestly, I think the perceived differences are all imaginary.
Right. I'm not a file system expert, but I'm increasingly wondering if your confident pronouncements are backed up by sufficient knowledge what you're talking about. Spotlight is implemented with major kernel support in the form of fsevents. This allows the user space portions of it to receive fairly reliable and timely notification of file system changes efficiently. This is a key ingredient to make it work as well as it does.
(see e.g. https://eclecticlight.co/2017/09/12/watching-macos-file-syst...)
Now the thing is, linux doesn't have a direct equivalent (or at least if it now has, it's a pretty recent thing, more than a decade after spotlight).
Quoting from lkml (https://lkml.org/lkml/2016/12/20/312)
Other operating systems have a scalable way of watching changes on
a large file system. Windows has USN Journal, macOS has FSEvents
and BSD has kevents.
The only way in Linux to monitor file system namei events
(e.g. create/delete/move) is the recursive inotify watch way and
this method scales very poorly for large enough directory trees.
> But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts?
Let me try again: with default settings my high spec linux box ground to unusable state (and no I'm not making it up) frequently enough that I got sick off it. So unlike you (somewhat rudely) continue to imply, I'm not some bozo who randomly screwed around with system settings he didn't grok on a whim and then started whining after everything broke.
> Again, you don't want that information in the file system table.
Yeah, you do because that way it stays around when you copy, move or archive the file. You probably only want to do that with a few select metainfo fields (like the examples I gave earlier: download origin info and user supplied tags), but that's exactly what macOS does. Also, whilst I agree that storing search indexes and everything directly in the file system is probably not ideal, there is historical precedent of a systems that did exactly that, fairly successfully from what i hear (BeOS/BFS).
P.S. maybe a more productive direction: what is you recommended way for setting up some baloo or some other linux indexer for running mdfind-style commandline queries with it (I don't want KDE or Gnome and I think baloosearch vs mdfind is also easier to compare directly)?
But that's not how any other those services work - including Spotlight.
> I'm not a file system expert, but I'm increasingly wondering if your confident pronouncements are backed up by sufficient knowledge what you're talking about.
I appreciate your frustration but the problem here is that you keep conflating multiple different technologies and not understanding the distinction I'm trying to make between each of them. I admit I'm not the best at explaining complex technologies (though I wouldn't say the stuff we're talking about is particularly complex) so maybe this conversation is better left to yourself to do some independent research because there is clearly a language gap between what I'm trying to describe and what you're apparently reading.
But the crux of it is you seem to think Spotlight stores all of it's data in the file system itself and is unique in that regard. That isn't true on both counts:
1. Spotlight will use a separate database - not xattr - to store it's indexes.
2. Every tool akin to Spotlight (including Krunner) does the same
There is the caveat that some of the searchable parameters in Spotlight obviously would be in the file system as well as Spotlights database - which might be where you're getting confused? But not everything you described would by xattr and Spotlight itself wouldn't be running slow file system scans to return it's results when it could instead use a local cached database (as I described above) with indexed fields against several parameters rather than just the inode number (which I'll get into later).
You also seem to think that inotify and/or fevents count as "file system integration". It does not. They are completely separate APIs. Whether they're backed by a kernel syscall is completely besides the point because they're not part of the file system ABI. Thus they're not actually tied to the file system itself (ie Spotlight can then work against any file system rather than just apfs).
> Let me try again: with default settings my high spec linux box ground to unusable state (and no I'm not making it up) frequently enough that I got sick off it. So unlike you (somewhat rudely) continue to imply, I'm not some bozo who randomly screwed around with system settings he didn't grok on a whim and then started whining after everything broke.
But you are over commiting resources to virtual machines then moaning when it grinds to a halt. Which isn't any better than tinkering with kernel parameters and making the same complaints.
> Yeah, you do because that way it stays around when you copy, move or archive the file.
That's what fevents is for ;)
By the way, even the file system doesn't index files by file name nor path. Every file system object (files, directories, TTYs, etc) on UNIX and Linux is just an inode. So even the file name and path itself is just metadata stored against the inode. The kernel itself doesn't understand file names, it just passes node indexes around and your file system driver will return metadata such as file name - if requested - by the calling userspace tool. That's how it works at a low level - even though file names and paths feel like a first class parameter in the userspace tools we use.
The reason you don't want too much metadata in the file system itself (eg xattr) is because it slows down file system operations. In fact many GUI platforms intentionally store extended attributes in hidden (technically just dot-prefixed because there isn't actually a "hidden" attribute on UNIX) for that reason. Partly that reason anyway - the other part is because not all filesystems support xattr. Which is actually another reason Spotlight wouldn't want to use xattr.
> Also, whilst I agree that storing search indexes and everything directly in the file system is probably not ideal, there is historical precedent of a systems that did exactly that, fairly successfully from what i hear (BeOS/BFS).
I did run BeOS but I can't remember much about BFS so I'm not going to comment on that specifically, however the other systems were split between two camps:
1. They either stored extended attributes in hidden files or directories - such as .Directory (KDE), .DS_Store (OSX), desktop.ini (Windows) - or
2. instead of a traditional file system layout they will have what is ostensibly be a fully fledged RDBMS. Those tended to be exclusive to mainframes but Microsoft was experimenting with a similar approach with WinFS in Longhorn (I think it was called?). However it was eventually canned due to it's shitty performance.
That's at least the hysterical precedence of storing super detailed meta-information. Historically the stuff that appeared to be stored as xattr were often just read from the file data itself (eg image sizes might be read from the JPEG headers). In fact in the 90s it was common for some platforms to identify what the type of file by literally reading the first few bytes of that file (eg does it have a pkzip header?) and some CLI tools still do this (eg `file` does exactly that. `grep` reads the first 1000 (exact number escapes me) bytes and if there is a null byte (0x00) then it is assumed to by a binary file rather than text and outputs an error.
As an aside, one of the hobby file systems I wrote was long the kinds of lines of (2) too. It used vanilla MySQL/MariaDB as the back end because one of it's features was that you could then connect to a remote filesystem via a simple MySQL connection string. It was a pretty fun project and I'd gotten all the read operations working but there was a few bugs with the write operations that I never fully solved and I eventually lost interest when I starting working other projects.
> P.S. maybe a more productive direction: what is you recommended way for setting up some baloo or some other linux indexer for running mdfind-style commandline queries with it (I don't want KDE or Gnome and I think baloosearch vs mdfind is also easier to compare directly)?
Honestly I don't know. I might not like Windows much as a platform but I do really like the explorer.exe shell as UI paradigm. So I tend to gravitate towards KDE on Linux (plus I think the KDE team have done a great job refining that paradigm in ways the Microsoft have failed to). Krunner has always "just worked" for me so I haven't spent any energy looking for ways to replace it. However I'm sure there will be some guides online about setting up runners (are they called?) on Linux given the diversity of it's ecosystem.
Back when I got my first SSD I ran Linux without a swap file/partition. I did this partly because it was only a 60GB SSD so I wanted to conserve space. I also did it because I didn't want to shorten the life of the SSD with (this was back when such a thing was a concern) and I ran that set up for years on a pretty modest 8GB RAM with KDE installed (ie not just a lightweight tiling WM).
> unsurprising since a proper version needs FS integration
It really doesn't and apfs (your file system in OSX) doesn't even do this. In fact it's probably better that your meta-data indexer isn't embedded into your file system driver because you're just going to slow down file system operations - which matters a lot on UNIX-based platforms because they do lots of file system operations.
A far better approach is to have your indexer run as a separate process that monitors file writes (you can still have a kernel hook for that if you wish) thus you can then catalogue your files without interrupting your normal file system operations. You can also add more granularity like separate database per home directory (which would be much harder to do securely if you were embedding that code into the fs driver without then going down the route of having multiple tanks ala ZFS). It also makes it much easier to optimize your meta-data db since you can now dump everything into a RDBMS rather than attached to the space constrained inodes.
For what it's worth, this is another area I have first hand experience with because I've written a few hobby file systems over the years. Nothing serious nor performant; just myself messing around with a few ideas. But it's still earned me a greater appreciation for the design decisions behind the file systems we do commonly use.