News for hackers. Stuff that matters.
Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @02:25AM (#12310731)
That's the thing about Linux: they have literally not one, but multiple armies of "programmers", and yet even after 20+ years of an absolutely furious development pace, basic things like inter-process communication still do not work desirably. Startup/shutdown? Yep, you know it already, it's still in flux (SMF in illumos has been humming along for a decade now). Filesystems? Flux. Observability? Flux. libc, what libc, that comes from GNU, flux. libstdc++, yeah we just decided to implement a "version 3 of that", flux. Debugging format? You have to rip out your runtime linker in order to get DWARF 2 support (DWARF 2 has been available for 23 years now), so, flux. Virtualization? Wait, which one of the 56 not-quite-right virtualization solutions do I mean, because Linux people are flapping with that and nobody can get it right? Flux.

The illumos kernel has been stable for decades now. Teams of five people here, three people there, four people over there, they roll up their sleeves, look at the issues the customers have had and are actually having now, professional engineers with formal education and insight, and in two years they have a working product which only needs a small fix here or there, and is tenable decades into the future. It goes to show that the number of programmers does not a quality, working product make.

For example: it's 2016, and my Linux filesystems in Vertica suddenly go into read-only mode, nobody knows why yet. In 2016! I cannot believe I just typed that, but it's true.

I've been running ZFS since 2006 both professionally and privately on Solaris and SmartOS at several large companies and I have never seen it go into read-only mode. The team which designed ZFS was only five people, not an army.

Bryan Cantrill summed it up so profoundly when, upon showing SmartOS running Linux applications, he ran /native/bin/dtrace and proclaimed:

I have clean water!!!

Re:Bus1: a new Linux interprocess communication proposal by vidarh on Thursday August 18, 2016 @04:41AM (#12311136)
It will always be in flux because it reflects what Illumos doesn't have: A userbase as diverse and large as Linux, of which a large proportion have competing needs and wishes and the ability to change it at will. This is not just about license either, but about the availability of people who understand the kernel.

I'm sure we could have something cleaner with a single, coherent team. But we also would have something that'd be useful to far fewer people.

Linux wins not by being best at everything. It has plenty of ugly warts. It wins by being accessible ranging from e.g. my ability to just take a job 20 years ago to port it to a custom x86 board with no bios or hard drive, because everything I needed was easily available, to the ability to hire any of a "an army" of past or current kernel developers to help solve my problems or do my custom development, to the ability for people to just write "yet another filesystem" and have actual real people try to use it and give feedback and contribute code.

The most important part of the success and value of Linux is that community. Without the community, it'd have little to offer over any number of other kernels that are better in some areas, worse in others.

But with it, it stands head and shoulder above all kinds of technically very impressive solutions because I can do things with and to Linux that I can't do, or can't do cheaply enough or in short enough time, with alternatives (and yes, that can be because of lack of knowledge of alternatives, but that too is part of the community/network effect).

Don't underestimate the importance of people.

> The team which designed ZFS was only five people, not an army.

And that's great. But the whole ZFS saga is also an illustration of how licensing early on can hamper adoption (KDE vs Gnome is another good illustration of how important license is; and how lasting the effect can be even after the licensing issues are sorted)

[dead]
Re:Bus1: a new Linux interprocess communication proposal by vidarh on Thursday August 18, 2016 @12:34PM (#12313977)
> Aha, you mean like redhat and IBM did, where there are now "kernel developers" who can't find their way out of a wet paper bag,

No, I mean like I can find any number of actual maintainers with a known track record and make them an offer to pay to have work done. Been there. Done that. Got good results.

> in order to have a look at what your presumption looks like in the real world...

It's not a presumption. It's actual experience in finding people to do work.

> ...Because we need more beta filesystems

Not everything is about need. While I work full time with Linux, a lot of what I use my computer for is fun, experimentation, learning. So yes: We need more beta filesystems. It's great that people experiment and learn. Everyone should try their hand at kernel changes, to learn - it's not magic.

> ...which ext3, which was on this Veritas disk group, wasn't able to recover, let alone detect. In the 21st century. Because community. I last saw a problem like this on Solaris ten years ago, since it has been a solved problem since 2006 there. Bryan is correct, I do have clean water, and even modern medical care facilities!

So use Solaris then, if you have so many problems with Linux. We're not forcing you to use Linux. It's not a dictatorship. I'll happily admit to all kinds of warts in Linux - I explicitly made the point that Linux has plenty.

> I have zero interest in how much of a community Linux has

Good for you. Meanwhile, for me, the community makes all the difference in making sure that I, e.g., has a decent sized hiring pool at all levels when I need developers, or which means that pretty much no matter what I try to do, there are people who have done it before I can ask questions of.

To me, that matters far more than avoiding a few warts.

Re:Bus1: a new Linux interprocess communication proposal by Nursie on Thursday August 18, 2016 @07:10AM (#12311571)
You exhibit a weird mixture of snobbery and incompetence. How odd.
Re:Bus1: a new Linux interprocess communication proposal by cyphar on Thursday August 18, 2016 @12:10PM (#12313689)
I've decided to stop interacting with Annatar, it's not worth the rage. Especially when he starts misrepresenting GNU/Linux technology that you are deeply into the development of (runC and containers in my case).
Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @05:30PM (#12316461)
Part of engineering a good solution is being able to recognize when you are in knee deep, and that it's not the best solution, and ditch it. That's what turns a good engineer into a great one, because then she or he turns around and builds the solution. You might not believe the grandpa here, but you're in knee deep, if not deeper.

On a personal note, I'm happy that you've stopped interacting with me. Good luck.

Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @08:05AM (#12311801)
Ah, my apologies for that: I did not realize that shoddy NFS software is due to my incompetence. I also apologize that I wasn't able to get something which kind-of worked in RHEL 5 to work in RHEL 6.5 because things which are completely unnecessary on UNIX (like worrying about separate portmapper service) and which were in userland are now suddenly in the kernel. But only for NFS 4. Because the documentation is actually unclear on whether that's the case for NFS 2 and 3. And because the NFS protocol specifies that client and the server should auto-negotiate which version they will use. Because of course the community can't figure out where what should be, so flux. Again.

I guess it's really snobby of me to except things which work on UNIX according to specification to behave the same way on Linux. Because Linux is cool. And it's great. And it has the biggest, bestest community. And because I don't appreciate just how great that is, instead being focused on getting work done with computers.

Now let's rehash inter-process communication again. Because it's fun rehashing things which work in other operating systems, over and over and over in Linux. Because Linux is cool like that.

Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @08:46AM (#12312048)
Don't underestimate the importance of people.

This warrants a reply of its own, since it is the core idea of my essays: it is the very difference between a stable, working, reliable product and Linux: the people. The right people. Competent people. People with the neccessary education, experience, and insight.

If if ever need heart surgery, I guarantee you that I won't go to the local shaman, no matter how happy he will be to perform the operation voluntarily and for free. Same goes for the local exorcist. If I ever need one, I'll get my surgery done by a specialist in cardiology, because they have been formally educated, formally trained, and have the requisite experience to operate on a human heart. And in the case of illumos / SmartOS, they will even do it gratis, like "medical doctors without borders" organization does. Right people for the job. For some reason, people here and at large believe that when it comes to inter-process communication, NFS, or writing operating system kernels anyone can do it.

Why exactly people believe that computer science is different from surgery, I haven't discovered yet, but based on empirical evidence of Linux and other crappy software, I have a hypothesis, and since the sample size is large, I might have a theory soon...

Re:Bus1: a new Linux interprocess communication proposal by vidarh on Thursday August 18, 2016 @12:20PM (#12313801)
But 99% of what we do are not that of heart surgery. Much of it is playing. Much of it is experimenting. Much of the actual work needs flexibility, familiarity and community more than it needs technical excellence.

Which is why I wrote what I did.

Your reply manages to turn it entirely on its head.

People believe computer science is different from surgery because it is: Nobody dies when my half-assed experiments with OpenGL crashes. Nobody dies if my laptop crashes.

There are certainly niches where it is life and death. Let others take care of that - I don't want anything to do with it.

Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @05:36PM (#12316492)
But 99% of what we do are not that of heart surgery. Much of it is playing. Much of it is experimenting.

And it shows. However, just because you can get something to work, in some way, does not suddenly make you (or anyone else) a professional. Case in point:

http://meta.serverfault.com/questions/6229/frustration-when-...

...Nor does it make your playful experiment fit for production where people like me are trying to get work done.

"I got something to work. Because Linux! Because community! Um... I think I'll go get a job as a computer professional now. What could possibly go wrong? Linux is so awesome!"

People believe computer science is different from surgery because it is: Nobody dies when my half-assed experiments with OpenGL crashes. Nobody dies if my laptop crashes.

An operating system's kernel is an extremely complex, special piece of software. A bug in such software can cause many people to die, which is why someday you might see disclaimers about running a nuclear facility and fitness of purpose. (Interprocess communication, the original topic, is part of an operating system's kernel.) Writing and diagnosing such software requires very deep insights, which very few have, even among computer professionals. Additionally, it absolutely and without fail requires formal education in the problem domain, it requires experience, and it requires apprenticeship, and I'm writing about physical one on one mentorship. It's not something one can pick up "from the community", never has been, and never will be, and if anybody tells you any different, they're either dangerously ignorant, or lying to you for the purpose of their own agenda.

Experimenting is all good and well, but when the entire armies of you start deploying your experiments in production and it becomes a trend and then the rest of us are, in fact, forced to take on such jobs because there aren't better ones, then it becomes a systemic problem. Not to mention that armies of you are currently in the process of artificially driving salaries down and it will be years before the market self-corrects, so make that a double whammy.

Now, me, I don't so much mind that cheap Linux labor is artificially driving salaries down, since cream always rises to the top, as they say, but I do so very much mind being forced by contract to work with things like ext3, which is based on concepts from the past century and which have since been long abandoned elsewhere, or that my really cool super slick database becomes unstable because of your experiment ticking underneath it (and before you ask, if it were up to me, I'd never put my database on your experiment). I mind even more when your (plural) experiements end up causing a priority 1 incident in the wee hours in the morning, when I'm busy getting my beauty sleep. In my case, you see, it's "because client contract!" Slightly different slant on things than in your case. Welcome to the industry.

Re:Bus1: a new Linux interprocess communication proposal by hawski on Thursday August 18, 2016 @07:37AM (#12311667)
You are probably right. But it sounds in very similar way to the Year of Linux Desktop in contrast to Windows Desktop or Linux Gaming in contrast to Windows Gaming. "Linux is all the better! But we don't have all the drivers, because of those pesky hardware companies. But we don't have all the software, because of those pesky software companies. But we don't have all the preinstalls, because of those pesky OEMs."

It might be better, but you can't strike out network effects. Because then you just end up with another rant why your thing is superior to that other thing everyone is using.

I am wondering if good old things are not just predestined to be unpopular. Probably only new things can get popular (even if they are just miserable copies of great old ideas). Next generation is more inclined to pick up something from the news than from history books. Next generation has their new lingo, so they will not understand easily old lingo. Next generation is cheaper to employ than older generation, so it's easier to get more in terms of quantity. Most (young?) people want everything, even if mediocre, now, than having few perfect things now, few perfect things later. The world belongs to the youth.

Re:Bus1: a new Linux interprocess communication proposal by qwertyuiop924 on Thursday August 18, 2016 @07:31AM (#12311643)
Well, you're feeling nice and smug, as usual. And after vomiting upon seeing this, I don't blame you.

OTOH, some of the above you mentioned isn't bad. Linux's "just a kernel" philosophy can be useful, and allows people to experiment with new inits (s6, runit, &c) and new systems that come in handy.

And go see Brendan Gregg's talk on Linux tracers. It's not as simple is dtrace, given the tracing war hasn't been won, but it's at a place where you CAN do most of the things dtrace does.

Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @08:57AM (#12312117)
Well, you're feeling nice and smug, as usual.

Yes I am, because I picked a highly reliable, stable substrate to build my own private infrastructure on.

Re:Bus1: a new Linux interprocess communication proposal by qwertyuiop924 on Thursday August 18, 2016 @09:53AM (#12312455)
Yeah, not arguing that, but you also have some really weird, misguided ideas. Like your understanding of bytecode. I think you just have a natural propensity for being smug.

As for using Linux, I'm not dropping Linux until LX-branded zones can run Steam, and my game library. And I don't see that happening for a while yet.

It may not be ideal for infrastucture, but if you want a UNIX on your desk, for all its flaws, Linux is pretty hard to beat.

Re:Bus1: a new Linux interprocess communication proposal by Annatar on Thursday August 18, 2016 @10:10AM (#12312555)
I think you just have a natural propensity for being smug.

I chuckled when I read that, and I want you to know that I appreciated it.

but if you want a UNIX on your desk, for all its flaws, Linux is pretty hard to beat.

I have Mac OS X for UNIX on the desktop. Works like a charm, and never gets in the way of what I'm trying to do.

Re:Bus1: a new Linux interprocess communication proposal by qwertyuiop924 on Thursday August 18, 2016 @12:46PM (#12314088)
That's fine for you. Mac has traditionally been a very good option. However, I hear Apple's been making it difficult in the lastest updates. For me, anyways, it's academic, as I can't afford apple hardware.
Re:Bus1: a new Linux interprocess communication proposal by Annatar on Friday August 19, 2016 @01:20AM (#12318116)
Yep, I understand that not everybody can afford it. To me, it's was a pure, dirty capitalist investment thing: that system paid for itself hundreds of times over with all the time and effort I've saved not having to mess with it. Best money I've ever spent, what with all the time I've saved, and all the jobs I obtained with it.

Have you looked at alternatives to GNU/Linux, like PC-BSD or FreeBSD? Did you find those inadequate / too much overhead?

Re:Bus1: a new Linux interprocess communication proposal by qwertyuiop924 on Friday August 19, 2016 @08:19AM (#12319506)
I already explained: As much as I love Linux for some reasons, and would love to move away from it for many others (broaden my horizons, escape systemd), I play video games. Right now, games barely get a Linux release. If I'm on BSD or Solaris, you can forget it. And while LX-branded zones are good, I think BSD is a better system for my needs, and in any case I doubt they're good enough to get the games talking to my GPU properly. And I'd love to avoid putting another layer between the games and the hardware if I have to. I remember using Wine.
Re:Bus1: a new Linux interprocess communication proposal by Annatar on Saturday August 20, 2016 @03:10AM (#12325560)
And I'd love to avoid putting another layer between the games and the hardware if I have to. I remember using Wine.

I agree, that makes perfect sense.

and in any case I doubt they're good enough to get the games talking to my GPU properly.

Now that's a whole different story: NVIDIA provides Solaris-native packages for their drivers, and they JustWork(SM). Way back in the day, NVIDIA engineers were stunned that they didn't have to produce a completely recompiled driver for every Solaris release: thanks to device driver interfaces/driver development kit[1], one could develop the driver on an ancient version of Solaris and have it work flawlessly on the latest-and-greatest one.

When I wrote that I'm using Mac OS X, that was the truth, but it wasn't the entire truth: I am typing this on my intel-based workstation I put together myself, with Solaris 10 running on it. And it has an NVIDIA GTX 980 accelerator, the top of the line model one could buy at the end of 2014. Downloaded the driver from NVIDIA's site, installed it with pkgadd(1M), and it JustWorks(SM)[2]:

  > pkginfo -l NVDAgraphics NVDAgraphicsr
   PKGINST:  NVDAgraphics
      NAME:  NVIDIA Graphics System Software
  CATEGORY:  system,graphics
      ARCH:  i386
   VERSION:  343.22,REV=2014.09.11.17.06
   BASEDIR:  /usr
    VENDOR:  NVIDIA Corporation
      DESC:  X and OpenGL Drivers for NVIDIA Quadro graphics
  INSTDATE:  Dec 04 2014 22:37
   HOTLINE:  Please contact your local service provider
    STATUS:  completely installed
     FILES:      188 installed pathnames
                  30 shared pathnames
                   2 linked files
                  42 directories
                   7 executables
              405482 blocks used (approx)

   PKGINST:  NVDAgraphicsr
      NAME:  NVIDIA Graphics System Device Driver
  CATEGORY:  system,graphics
      ARCH:  i386
   VERSION:  343.22,REV=2014.09.11.17.06
   BASEDIR:  /
    VENDOR:  NVIDIA Corporation
      DESC:  Kernel Drivers for NVIDIA Quadro graphics
  INSTDATE:  Dec 04 2014 22:37
   HOTLINE:  Please contact your local service provider
    STATUS:  completely installed
     FILES:        6 installed pathnames
                   3 shared pathnames
                   3 directories
               52151 blocks used (approx)
However, all of that is not to say you could play games on Solaris. If they were really optimized for Solaris, then they would fly, as Solaris is really, really, really fast, especially on intel. But since they aren't, and that's not realistic right now (and might never be realistic), I completely understand where you're coming from.

As for Mac, OS X has plenty of major and indie titles in the Apple app store. But, it's for people who are actually willing to pay for that.

[1] http://docs.oracle.com/cd/E19253-01/816-4854/

[2] http://www.geforce.com/drivers/results/105347

Re:Bus1: a new Linux interprocess communication proposal by qwertyuiop924 on Saturday August 20, 2016 @10:13AM (#12326692)
I was really concerned with a) driver perf, and b) how the drivers worked in an LX-branded zone. Steam and the Humble Bundle will leave me set for games.

And now, because you didn't ask for it, Game Reccomendations:

Have you tried out TIS-100? You can get it through Steam, and maybe a few other places, it runs on Mac, and I'm willing to bet it'll run in an LX-brand. It bills itself as "the assembly language puzzler nobody asked for," and has a focus on minimal instruction count, maximum speed, and paralellism. You should know by this point if you're interested.

You also may want to check out Xonotic, which is an arena FPS with a relatively lively community, which may compile on Solaris. Think a mixture of Quake and UT.

Re:Bus1: a new Linux interprocess communication proposal by Annatar on Sunday August 21, 2016 @08:18AM (#12330650)
I didn't ask for it, but I love the clues you drop... "TIS-100"? "Xonotic"? My interest is further piqued, now I have to go and research...

Apropos Qt, it is not for the faint of heart: even on platforms it officially supports, it is very hard to build, at least it was last I had to do it professionally, back in 2013. Qt is really unprofessional as a product, I remember it involved a lot of hacking and coercion of the build engine to get it to build. And it's monstrously large.

Post scriptum:

ah, "Xonotic" is a fork of "Nexuiz". Okay, that's clear now. "TIS-100" is a puzzle game. So these are individual titles, not technology which enables running on a different operating system. "The Ur-Quan Masters" used to build and run superbly on Solaris; it showcased just how good of a gaming platform Solaris could be if the code is good and clean.

lx-brand could be made to work, this screenshot shows it's possible:

https://www.perkin.org.uk/posts/whats-new-in-pkgsrc-2013Q2.h...

I haven't researched how to translate that into the lx-branded zone though (what the package equivalents would be), primarily because I viewed even the above as too much work: for example, the desktop would have to be brought over to somewhere with an X server. I don't see how it could be made to start on the bare metal's display when it's running in one of the hypervisor's zones?

Re:Bus1: a new Linux interprocess communication proposal by qwertyuiop924 on Sunday August 21, 2016 @02:04PM (#12331948)
Yes, that's why I said game reccomendations. I've been meaning to check out UQM for a while now.

Xonotic should run on Solaris. DarkPlaces, the engine codebase, is fairly portable last I checked.

TIS-100 runs atop Mono, but it's probably not using GPU heavily, so it'll probably run in an LX brand if you can get X up and running. As for running X11 on LX, the answer is both simple in its phrasing and hellish in its implications: X Forwarding.

I kid. Although it does seem like X forwarding is the easiest option. Luckily, you're not forwarding over the net, so you should be fine.

But yeah, it's nice to have a simple discussion about running video games in ways not intended, and what games are good, after spending a while explaining why systemd is a bad idea for the 15th time, because some people are still taken in by the sparkly unit files, and aren't paying attention to the beast lurking below the surface. (If such people are reading this thread, I'll give you a hint: Separation of concerns is a good thing, and if the process will panic the kernel if it crashes, I want it doing the job it's required to do. AND NOTHING ELSE.)

Re:Bus1: a new Linux interprocess communication proposal by chme on Thursday August 18, 2016 @04:22AM (#12311055)
> Flux.

And while all that is constantly changing, its still possible to compile very old user space software and run it in on a current kernel.

Re:Bus1: a new Linux interprocess communication proposal by IshKebab on Thursday August 18, 2016 @07:38AM (#12311677)
Yeah but is it possible to run very old binaries on a current kernel? Unlikely. What about binaries compiled on a newer system than your own? Even less likely.
Re:Bus1: a new Linux interprocess communication proposal by majewsky on Thursday August 18, 2016 @04:31AM (#12311087)
Well yeah, most of the libc API is older than Linux.
Re:Bus1: a new Linux interprocess communication proposal by ekidd on Thursday August 18, 2016 @07:00AM (#12311549)
The Linux libc API changed a couple of times in the 1990s (a.out → ELF, libc5 → libc6, etc), but the Linux kernel ABI (what libc uses to talk to the kernel) has been stable since practically the beginning. Linus detests breaking anything that talks to the kernel and considers it massively unprofessional, to the point that even a terrible API will be set in stone if anything uses it.
Re:Bus1: a new Linux interprocess communication proposal by Annatar on Wednesday August 24, 2016 @02:50PM (#12354317)
What about drivers?

On SunOS, I can install a driver built on SunOS 5.5.1 (Solaris 2.5.1) on the latest illumos or Solaris-based OS, without recompilation, thanks to the device driver interface / device driver kit ("DDI / DDK"). Solaris 2.5.1 came out in 1993; illumos came out, well, today (multiple commits every day).

Is the same possible on GNU/Linux?

Re:Bus1: a new Linux interprocess communication proposal by jamespo on Thursday August 18, 2016 @04:55AM (#12311192)
ZFS/SMF/zones etc are due to the engineering excellence at Sun though, I'd be interested to know of the innovation/improvements since Oracle slammed the door on OpenSolaris.
Re:Bus1: a new Linux interprocess communication proposal by RantyDave on Thursday August 18, 2016 @06:35AM (#12311475)
You can now start a zone that's compatible with 64-bit Linux, or one that runs a "hardware" hypervisor, or mix and match the two. That sort of thing?
Re:Bus1: a new Linux interprocess communication proposal by jamespo on Thursday August 18, 2016 @09:37AM (#12312361)
Yes, that's interesting. Not on a par with the DTrace or ZFS projects but an incremental improvement all the same.
Re:Bus1: a new Linux interprocess communication proposal by Annatar on Friday August 19, 2016 @01:13AM (#12318087)
Not only is it on par, it's revolutionary. I'm referring to imgadm(1M) and vmadm(1M) in particular.

The guys didn't have to get out of Oracle to reinvent the wheel: the engineering they did is sound and tenable enough for decades into the future, so they built upon it.