News for hackers. Stuff that matters.
Re:Darling – macOS Translation Layer for Linux by laumars on Monday April 29, 2019 @11:51AM (#19779472)
I don't mind OSX - honestly there's bits of it I do genuinely like. But it's blind-sighted fanboyism that really does the platform harm.

> keybindings

Honestly, I find Mac OS keybindings and keyboard layout the worst thing about using Macs. Yes it might make some sense but when literally every other platform on the planet follows the same standard apart from Apple, it then makes Apple the ugly stepsister regardless of how rational it might be on paper.

I mean if you only ever use Macs then I guess you might like it, but for anyone who swaps between systems (or even just wants to use a non-mac keyboard) it can be very annoying.

> file history

You can have that in Linux

> Cmd-? (how do I do that on Linux?)

shrugs maybe you can't. But that's just one feature. As a counterargument I could list a dozen things that are in Linux that aren't in OSX. Like up to date core utils and proper package management. Which are just about the two most important things on a dev machine - far more important than Cmd-?. And sure you could install GNU core utils via brew, none of that is part of the default OS X build - which matters because the whole basis of your argument was that OSX has better defaults.

Ultimately though I don't see the point in nitpicking each OS - feature by feature.

> finding files

I'd already disagreed this in my previous post after you made that claim earlier

> MacOS can recover from memory pressure fine. How do I get my linux machine not to effectively crash if I run an app that happens to use too much memory (technically it just swaps itself to death, in practice reset is the only remotely timely way to recover)? I've tried any amount of tweaking, but turns out that you can't turn off overcommit and swap completely, even if have lots of memory, things will just break randomly (chrome for example).

The problem there is the application. However Linux will just kill the last process that over allocates memory. If you're getting the kind of symptoms you've described then you've either fiddled with your swap file settings (so not running defaults) and/or you're running Linux on some pretty awesome spinning disks while comparing it to nice fast SSDs on OSX. Either way, you're not comparing like for like.

> Can you point me at something on linux that comes anywhere close to spotlight/mdfind?

There's loads. Krunner, for example, has all the same features as Spotlight plus supports plugins to extend it. For example I can run math calculations in it - which I haven't yet worked out how to do in Spotlight.

> For what it's worth: I'm using both linux and macOS daily and am both productive and reasonably expert with both.

But do you actually use desktop Linux on modern hardware? Or are you just running Linux on a few servers and guessing about the desktop experience. I ask because your comments were valid about 10 or 15 years ago but really aren't the case any longer.

> That is not to say that linux isn't more ergonomic for certain things, but in my experience they tend to be mostly limited to things only programmers would care about

This I do wholeheartedly agree with.

Re:Darling – macOS Translation Layer for Linux by patrec on Monday April 29, 2019 @07:40PM (#19783695)
> > file history > You can have that in Linux

You can use lvm or zfs snapshots, but that's not what I'm talking about – I'm talking about in-app browsable history of things like documents or presentations.

> Krunner, for example, has all the same features as Spotlight

Last I checked it used Baloo to do the actual indexing, the list of high priority features/bugs on the project site https://community.kde.org/Baloo (Baloo crashes a lot in various places etc.) and a quick google make it look like it remains alpha software at best, I'm also pretty sure it doesn't have an equally reliable index update mechanism. The most important thing about spotlight for me is that it can search file names and content (filtered by type if necessary) fast and reliably and up-to-date. But you can also do types of searches that as far I'm aware of none of the linux utilities can do.

E.g. show me all the items I downloaded from a google.com domain:

    mdfind "kMDItemWhereFroms == '*google.com*'"
If I want to see the filetypes that where downloaded sorted by frequency I can add

    mdfind -0 "kMDItemWhereFroms == '*google.com*'" | xargs -0 -n1 mdls -name "kMDItemWhereFroms" | sort | uniq -c | sort -n
> For example I can run math calculations in it - which I haven't yet worked out how to do in Spotlight.

You literally just type what you want calculated, e.g. `sin(pi/4)`.

> As a counterargument I could list a dozen things that are in Linux that aren't in OSX. Like up to date core utils and proper package management.

nix. By my lights the only proper package management for any OS. Works fine under both linux and macOS (and will also trivially supply you with up to date coreutils).

> But do you actually use desktop Linux on modern hardware?

I have been using (well-specced) linux desktops for most of my work for a long time.

> The problem there is the application. However Linux will just kill the last process that over allocates memory.

I don't think that's how it works. The whole point of having a proper OS (rather than say DOS) is that misbehaving app won't just bring down everything else. Also if you have a process that wildly allocates memory, by default linux will start off swapping like mad, making your computer effectively unusable (and yeah, in fact my linux desktop does have an SSD and many times as much RAM as my macbook, so if I'm not comparing like to like my linux station is the one with much more powerful hardware). And even if it runs out of swap it doesn't just kill the last process, it uses a more complex scoring algorithm which has a good chance of killing something you didn't want to be killed.

Re:Darling – macOS Translation Layer for Linux by laumars on Monday April 29, 2019 @08:34PM (#19784014)
> You can use lvm or zfs snapshots, but that's not what I'm talking about – I'm talking about in-app browsable history of things like documents or presentations.

I got that. It's still just some application UI wrapped around a CoW file system. Maybe a better way of saying your point is "doing the same on Linux lacks a lot of polish" - which is true. But that's when happens when Linux has to support a multitude of file systems but Apple can control every aspect of their ecosystem.

> Last I checked it used Baloo to do the actual indexing,

Possibly? Krunner has always "just worked" for me so I've never bothered to look under it's hood.

Regarding the bug you found, well I'd argue that you should expect to read bugs on a bug tracker given that's the point of bug trackers. It does feel like what you're basically now doing is the equivalent of reading a 1 star review of a product (eg on Amazon) and claiming it doesn't work by the proxy of others while also ignoring all the 5 star reviews from people who haven't had any issues. It's a heavily biased way to hold a discussion and if we're both honest, Mac's haven't been without their fair share of bad publicity either. So is it really worth our time cherry picking all the negative things when you and I both know that they're the exception rather than the norm?

> You literally just type what you want calculated, e.g. `sin(pi/4)`.

Handy to know. I suspected it would have been possible but I kept prefixing the formula with `=` which Spotlight didn't like.

> nix. By my lights the only proper package management for any OS. Works fine under both linux and macOS (and will also trivially supply you with up to date coreutils).

My point is you shouldn't have to install a 3rd party package manager. That's the bare minimum a modern OS should provide out of the box.

> I have been using (well-specced) linux desktops for most of my work for a long time.

I struggle to believe that given the descriptions of faults that you've been discussing. Though you have also said you've tinkered with the "swappiness" parameters (plus more) so I guess it's possible that you are running current hardware but have inadvertently tweaked Linux into performing terribly? Or maybe you're just exaggerating all these problems to make a point (much like your "look, I've found a bug on a bug tracker" comment above).

Either way, if the problems were as prevalent and severe as you keep describing then you and I - and millions of other techies for that matter - wouldn't be running Linux.

> The whole point of having a proper OS (rather than say DOS) is that misbehaving app won't just bring down everything else.

"Proper OS" is such a flakey term and what you described isn't even the "whole point" of running an OS. But that's a whole other tangent. More importantly Linux doesn't do what you're accusing it of doing. Thus your statement is simply untrue in a multitude of ways.

> Also if you have a process that wildly allocates memory, by default linux will start off swapping like mad

It's actually a great deal more complicated than that. It depends on the size of your swap file, what applications you have open and their current running state (ie can they be paged). It depends on whether your cache is non-zero and it also depends on the kernel parameters you define.

> And even if it runs out of swap it doesn't just kill the last process, it uses a more complex scoring algorithm which has a good chance of killing something you didn't want to be killed.

Depends on the version of Linux (the kernel) you're running. Older kernels will just kill the last requester. Newer kernels do have a scoring algorithm but it's really not that complex at all (if memory serves, it's ostensibly a just a percentage*10 figure of used memory)

Re:Darling – macOS Translation Layer for Linux by patrec on Tuesday April 30, 2019 @02:50AM (#19785682)
> It's still just some application UI wrapped around a CoW file system.

So? In terms of usability impact I still consider it a major feature (that no amount of tweaking will get you on Linux).

> My point is you shouldn't have to install a 3rd party package manager.

But macOS has a "package manager" – it's called App Store. You and I may not think it sufficient for our (developer) needs, but we're not representative users. And for normal users and even myself it offers very useful functionality over what they'd get out of the typical native linux package manger. You can trivially reinstall everything on a different machine with a different os version (as long as it's not super ancient), and it works – no "DLL" hell, because everything is essentially self-contained. And since software is tied to your account, there is no need for crufty apt queries in the hope to get a list of packages out you can backup for reinstall elsewhere or after clean upgrade. Ubuntu has tried to establish a clone in the Snap Store, but no one I know seems to use it and I haven't tried it myself, so I don't know how compelling it is.

And I need to install a 3rd party package manager on (non-NixOS) linux distros anyway, because IMO apt, yum etc. fundamentally suck and nix is the only thing that doesn't. Funnily enough, the only really compelling UX argument for linux instead of macOS for developers I can think of apart from /proc is that with NixOS you can codify your complete machine setup in a single nice config file, making it super easy to replicate, backup or inspect.

> Or maybe you're just exaggerating all these problems to make a point [...] More importantly Linux doesn't do what you're accusing it of doing.

It's a bit annoying to be told that what I'm saying literally can't be true. It is, and I didn't tweak any sysctl params or the swap setup before I got tired of my machine grinding to a halt and me having to reset it. I can ensure you it's entirely possible to have a high grade desktop with SSD and have linux fall over swapping endlessly without even being able to move the cursor anymore. Of course this doesn't happen in "everyday" usage otherwise no one would be running linux, but it's not that hard to trigger if you're running a VMs, a few browsers and dev tooling stuff that can potentially consume large amount of memory very quickly. I've moved away from having to use these tools (and also tweaked my machine) so it hasn't been a problem off late, but I ran into it with completely stock ubuntu.

Re:Darling – macOS Translation Layer for Linux by elderK on Wednesday May 01, 2019 @05:28AM (#19795658)
Drive-by poster here but I was wondering if you had spent any time looking into what IO scheduler you're using on Linux?

Some time ago I encountered issues similar to what you mention in your posts. I solved it selecting the "Deadline" IO scheduler when I built my kernel.

Hopefully this helps you solve the issue :)

~K

Re:Darling – macOS Translation Layer for Linux by laumars on Tuesday April 30, 2019 @07:36AM (#19786993)
> (that no amount of tweaking will get you on Linux).

It best not to use such firm definitives like that when what you actually mean is "more tweaking than a typical user would be bothered with". :)

> But macOS has a "package manager" – it's called App Store. You and I may not think it sufficient for our (developer) needs, but we're not representative users.

You're seriously going to defend the App Store?! The App Store isn't just garbage for developers, it's garbage for everyone because it misses so many non-developer productivity tools too. It doesn't even have Chrome nor Firefox in it.

> And for normal users and even myself it offers very useful functionality over what they'd get out of the typical native linux package manger

Sorry but I'm not buying that argument. You claim to be a "normal user" then talk about messing around with kernel parameters in Linux. I really don't think you're making any fair and balanced arguments on this topic at all.

> You can trivially reinstall everything on a different machine with a different os version (as long as it's not super ancient), and it works – no "DLL" hell, because everything is essentially self-contained.

I guess if you compare the App Store to manually loading software on Windows - literally the worst platform ever created for managing installed software - then the App Store would look good. But likewise if you compare a heart surgery to a lobotomy then heart surgery would look less invasive too. This is why I don't think it's productive to compare solutions to the worst examples.

> And since software is tied to your account, there is no need for crufty apt queries in the hope to get a list of packages out you can backup for reinstall elsewhere or after clean upgrade.

It's a pity that the App Store offers so little software so you end up falling back to 3rd party package managers. So now on OSX you not only need to run the same "crufty [package manager] queries" on OSX but you also need to install the package manager itself too.

> And I need to install a 3rd party package manager on (non-NixOS) linux distros anyway, because IMO apt, yum etc. fundamentally suck and nix is the only thing that doesn't.

A moment ago you were claiming to be a "normal user". Normal users don't install nix :) tbh I'm not the biggest fan of apt, yum is better but I do really like pacman. However claiming apt and yum suck when also praising the OSX App Store is just weird.

> I can ensure you it's entirely possible to have a high grade desktop with SSD and have linux fall over swapping endlessly without even being able to move the cursor anymore. Of course this doesn't happen in "everyday" usage otherwise no one would be running linux, but it's not that hard to trigger if you're running a VMs, a few browsers and dev tooling stuff that can potentially consume large amount of memory very quickly.

Right, I get you now. That context helps. Your previous description just said you were running a browser and sounded like it was happening everyday (so basically you were exaggerating by leaving key details out when describing the root cause). The problem there is that you're not just over commiting on memory but also over commiting on CPU resources too. That latter part matters because swapping can be CPU expensive too. Hence why your system is grinding to a halt.

Also I still think you're to blame a little there because if you're running VMs then you should be setting their thresholds to a level that doesn't overcommit your systems resources (baring in mind these tools aren't the stuff that "normal users" would be using either). It's like opening a bottle of wine and pouring yourself 4 glasses then complaining that the bottle is empty and you couldn't squeeze out a 5th glass (can you tell I'm drinking wine at the moment hehe?). You only have a finite amount of system resources so you cant really complain if you intentionally over commit them.

Re:Darling – macOS Translation Layer for Linux by patrec on Tuesday April 30, 2019 @10:13PM (#19793938)
> You're seriously going to defend the App Store?!

Yup, flawed as it is, I find it much more useful than apt. If I'm wearing a dev hat and were forbidden from using anything to manage software installs other than one of apt or App Store (no nix!), I'd rather have apt. But for my non dev apps (you know, even people who tweak kernel parameters have non-programming related apps they want to use from time to time ;), Appstore is obviously more useful.

> However claiming apt and yum suck when also praising the OSX App Store is just weird.

Why? Both fill different needs and App Store solves a problems that are useful to me acceptably well (making it easy to install up-to-date software I want, upgrade it and remember what I have on a per-account not per machine basis).

Yum and apt, on the other hand don't (they don't have up-to-date software I want, they don't give me what I consider a decent way to manage the same or similar setups on multiple machines etc.). I basically install everything I can with nix instead.

> So now on OSX you not only need to run the same "crufty [package manager] queries" on OSX but you also need to install the package manager itself too.

Unlike apt/yum nix offers good ways to do this – no cruftiness involved. E.g. you can just write a small file with what you want and you'll get it, on any machine.

> You only have a finite amount of system resources so you cant really complain if you intentionally over commit them.

That's not what's happened, my VMs where capped at reasonable limits. I used to run some tools for various reasons that could in some scenarios eat a lot of a ram fairly suddenly (I don't think the systems was anywhere close to overloaded CPU wise was true most of the time but I can't vouch I remember this right anymore).

Either way, I don't think the whole OS falling over because one app wants to consume too much memory and the OS has decided to never say no is reasonable. And it's not something I can recall ever happening to me with any other OS (in recent years, I don't want to think back to ancient windows days).

Re:Darling – macOS Translation Layer for Linux by laumars on Wednesday May 01, 2019 @04:06AM (#19795324)
> they don't have up-to-date software I want

I take your multi-machine point but the above just depends on what repository you're pointing at (eg stable, testing, etc) and which Linux distro you're running. You can't really blame apt for being out of date if you're running Debian. And nor could you blame apt for delivering buggy packages if you're running the testing repos on Ubuntu.

It's the same package manager, just different end points.

> Unlike apt/yum nix offers good ways to do this – no cruftiness involved. E.g. you can just write a small file with what you want and you'll get it, on any machine.

Technically you can do that with any package manager - given that's the core point of a package manager :P

I've not used nix (read a little about it but never taken the time to try it) so I can't comment how much easier that makes the process of custom repositories than hosting your own apt or yum repo, but it's not actually hard to do in those two either. Plus you could always compile your own .deb or RPM and install it like a standalone installer (MSI et al).

I've got nothing against nix though. In fact weirdly I think your underselling nix by focusing on the points you have rather than it's major differences from traditional package management.

> That's not what's happened, my VMs where capped at reasonable limits. I used to run some tools for various reasons that could in some scenarios eat a lot of a ram fairly suddenly (I don't think the systems was anywhere close to overloaded CPU wise was true most of the time but I can't vouch I remember this right anymore).

The problem with over commiting is the limits might seem reasonable under normal workloads but when you do end up with an empty bucket you have no safe way to recover from that. Or at least not with desktop virtualisation solutions like VirtualBox. ESXi et al will handle such situations more gracefully because they're designed to over commit during off peak work loads.

That said, I don't know how long ago it was you last did this but a few years ago VirtualBox did add a CPU execution cap in the guest config. IIRC it defaults to 100% but if you're running multiple guests and/or running heavy applications on the host while also running heavy guest VMs then it's worth dropping the CPU execution cap down so the guest cannot lock up the host.

> Either way, I don't think the whole OS falling over because one app wants to consume too much memory and the OS has decided to never say no is reasonable.

I think your expectation here is a little unreasonable to be honest. You cannot drain the host of free system memory and idle CPUs then expect the host to gracefully recover. It's like trying to dowse a fire with an empty bucket. I honestly can't see how OSX would perform any different to Linux in that regard. So you were probably using different virtualisation technologies on OSX (VMWare perhaps?) that handle guests more responsibly.