Very interesting... my first (and last) encounter with heterogeneous OSes was back in the mid nineties with TaOS (best reference found online is http://www.uruk.org/emu/Taos.html but the website is rather wonky).

Honest question: I’m not the only one who confused this for the Popcorn Time torrent streaming player, am I?

My first thought was: wow, they took this idea so far now they made an OS out of it.. I imagined an OS preconfigured with torrent clients, easy-to-use VPN client, fancy Netflix-like media center...

You're not!

An announcement from a few days ago was posted here: https://news.ycombinator.com/item?id=23060692 (no comments though)

A thread from 2017: https://news.ycombinator.com/item?id=14605882

Can someone ELI5 what this is?

It's a single system image [1] platform built with Linux. The main idea being that the OS is running on multiple computers while presenting a single computer abstraction to the other software running on it. You write your program as though it's running on a monolithic machine, and the SSI platform manages it across multiple distributed machines.

The concept is more popular in HPC & scientific computing than it is in mainstream IT.

[1] https://en.m.wikipedia.org/wiki/Single_system_image

You know, this is a great, clear description that they should definitely put on their homepage.

Great explanation; I can't stop watching the teaser http://popcornlinux.org/sources/video/sosp-19-tutorial.mp4

This is off topic, but it's proving hard to google. What's the significance of the hand gesture the women in the video makes at 1:13 and again at 1:39?

It is a prevalent hand gesture in the Indian subcontinent. Could mean a lot of things, and interpreted along with the facial expression. In this case, it means "That's pretty cool", looking at her facial expressions.

It could mean "What the heck is going on?", when accompanied with a different facial expression.

Exactly the same in Italy. Weird, that.

Surprisingly, as an Italian, I know exactly what this means: “wonderful, amazing, perfect”.

as an Italian, I understand this as "to steal"

I’d denote stealing with my palm facing downwards.

I actually had to try it out in real life... I can’t think of gesticulating in the abstract.

Seems like you can take the Italian out of Italy but not Italy out of the Italian.

It means "wow, impressive".

looks like that clip of her is looped, so it might just be a random hand movement that looks meaningful bc you see it twice

wow, i sure feel dumb about this now! thank you, sibling commenters who actually know stuff :)

I was far more distracted by the adorableness of very enthusiastic people that aren't really skilled at reading off a teleprompter than their accents.

Being able to understand ESL speakers seems like an increasingly important skill if you work in tech, was it really that hard for you?

I judge the video not by the enthusiasm of the people involved, but by the actual contents of the video.

You know what is an important workplace skill (tech or not)? Speaking intelligibly.

Some people seem to think that all criticism of accents is bad, because racism. This is actually backwards. I myself am an ESL speaker, and therefore have some trouble understanding heavy accents. Using people with heavy accents in a video instead excludes people who speak English as a second language, since we, as a group, are not trained in it. Our experience of spoken English have mainly come from TV and movies, where people speak clearly. So when we see something like this video, we are effectively excluded from understanding it.

Saying things like “was it really that hard” is condescending. Yes, it’s hard if you don’t have a lifetime of training, which people in English-speaking countries do. We ESL speakers don’t. Producers of videos should be more considerate.

I'm not sure that's entirely accurate. I know I have an easier time understanding History on Fire[1]'s thick Italian accent than Americans do. When learning English, you learn the correct pronunciation and also the "incorrect" pronunciation used by your classmates.

[1]http://historyonfirepodcast.com/

None of the speakers in the video use my local version of what passes for English, and therefore I have no training in understanding the video. It is difficult for anyone who has not lived with people having all of those accents. Which is unlikely if you’re living in a non-English speaking culture.

I can't agree more. Outside of the US you're interaction with ESL speakers varies wildly but it's a pretty essential skill nowadays.

I'm the only native English speaker at my company. Some folks speak perfectly, others don't.

The people in this video spoke though, even if they did have a fairly heavy accent.

> Outside of the US you're interaction...

* your interaction

Sorry, couldn't resist correcting it on a comment about interactions with ESL speakers :)

They speak perfectly intelligible English, you just need to listen for a second to understand the accent. Subtitles would be great but come on, there's no reason to be a jerk about it.

Sounds a bit like a Beowulf Linux cluster from about 20y ago. What's the major difference between the approach of those.

Beowulf had no shared memory abstraction, programs had to be written using message passing, e.g. MPI.

I'm not convinced a shared memory abstraction is a good thing for distribution across machines. Some Beowulf clusters actually had much lower latency interconnects than ethernet, e.g. Quadrics, but we still needed to carefully optimise the traffic. With a shared memory abstraction, such optimisation is out of our hands.

Beowulf was just the idea of making a cluster of cheap commodity hardware using open software, typically not Single System Image.

Yeah I was thinking this sounds more like MOSIX, but with the added abstraction to handle executing the same code accross different architectures.

http://www.mosix.cs.huji.ac.il/txt_about.html

Indeed, MOSIX or Kerrighed http://www.kerrighed.org/wiki/index.php/Main_Page

Ah! I've been saying we should do this for years now. Everyone in the cloud space is reinventing the wheel when what we need is a cloud OS that abstracts away multiple computers. I can see a bright future for a project like this.

I can't find it but that sounds like the model a platform company that works with CCP to host EVE online makes. They program one game but they shard different sectors of the universe out to hardware based on demand.

They either host it or it was just a tech demo they did with eve online. Wish I could remember it was a super cool job advert, not my skill set though.

In mainstream IT, you'd usually set up a Hadoop or Spark cluster on the machines and then write your code to run against one of those. What's the advantage of this approach over those?

Automatic vertical scaling without changing applications. One needs to rewrite apps in order to get the advantages of Hadoop/Spark/MapReduce in general, and not all problems are solved by simply doing that.. It is a different thing/paradigm, really

Does this mean I can expand my thinkpad with one or more raspberry pi boards?

aka Linux Cluster. This was very popular concept about 15yrs ago. I have seen telecom servers run in such kind of clusters

Are you referring to servers running OTP (erlang)?

You mean the intro paragraph isn't simple enough?

> The Popcorn Linux project is exploring how to improve the programmability of emerging heterogeneous hardware, in particular, those with Instruction Set Architecture (ISA)-diverse cores, from node-scale (e.g., Xeon/Xeon-Phi, ARM/x86, CPU/GPU/FPGAs) to rack-scale (e.g., Scale-out processors, Firebox, The Machine), in both native and virtualized settings. Additionally, the project is exploring how to automatically compile/synthesize/execute code on ISA-heterogeneous hardware.

I know some of those words!

Let's break it down! (I'm no expert so sorry for any mistakes, and this is simplifying a lot)

> improve the programmability of emerging heterogeneous hardware

reducing the fiddlyness of developing for mixed hardware

> those with Instruction Set Architecture (ISA)-diverse cores

those with multiple CPUs speaking different machine code

> in both native and virtualized settings

both bare-metal and VMs

> automatically compile/synthesize/execute code on ISA-heterogeneous hardware

automatically translate between the different machine codes.

---

Now for what it actually means, is that Popcorn is a single Linux that can simultaneously run on a bunch of different processors (say, your mobile phone, a raspberry, and a desktop) all at once while being used just like a single Linux system to the end-user/programmer. I don't think it's meant to be used by consumers, but for servers or supercomputers, that kind of stuff.

Pairing a big fast processor with a small energy-efficient processor is starting to get more common. You see it in a lot of higher end Android phones.

I believe that the Touch Bar Macbooks also pair an x86_64 processor with an ARM coprocessor that runs the touch bar.

> I don't think it's meant to be used by consumers, but for servers or supercomputers, that kind of stuff.

Why would servers/supercomputers care about heterogeneous hardware (including multiple ISAs, even!)? If I'm playing around at home, that's great; I'd love to make a cluster out of a couple of old laptops, an old phone, and a Raspberry Pi. But at work, I'd just... buy a dozen identical servers. Even across generations, x86 is mostly uniform enough to just ignore the changes (with some minor gotchas around extensions, but if you only migrate forward that should be fine too).

A supercomputer cluster wouldn’t care. A cluster of supercomputer clusters might.

Know those moments in movies where they put “all the computers in the world” onto the task of cracking some encryption before it’s too late? This could be used for the technical part of that (once you’ve got all the HPC labs’ cooperation.)

Could an abstraction of "single Linux computer" be actually useful with Earth-scale latencies?

Think of it in a "Compute at home", "folding at home", blockchain-sense then latency doesn't matter as much, I do not know if anyone has tried solving things with big clusters separated by 40ms-60ms latencies.

Maybe if you had some kind of amazing cache system that organized everything into one big tree so that data needed for different computations is as local as possible. It would be very challenging though.

at Cray, at one point we had custom vector processors, custom SMT(for lack of a better term) processors, FPGAs on the roadmap, GPUs, and commodity AMD processors...and a strong desire to ship them in the same frame

that may have been the only time in history though

Some ISAs can be way more effective at some parts of the problem than others, while still not being an optimum for the entire problem.

That's why your computer has a GPU.

Can it run on a gpu so to abstract always cuda

I imagine it can't (that would be way too good, everybody would be using it already). I also don't think there are many examples of supercomputers with diverse ISAs today.

But a system like this is a requisite for non-uniform computers to exist. Thus it must exist without the other, at least for a while.

I think I bingo'ed at least 3 times.

It looks like a multi-node version of Linux - a bit like a MOSIX or something - but without the limitation that the hardware needs to be the same on each node. E.g. you could have one X86, one ARM, one FPGA, and a single app could deploy threads across each of those "CPUs".

Equally, I guess it looks a bit like a Java application server in terms of function - applications appear to be compiled to whichever hardware target the application needs to run on.

Utility infrastructure like shared memory etc appears to be part of the design, so the underlying hardware systems are as transparent as possible.

Name brought back fond memories of SLAX Popcorn Linux. 128MB vs ISO's today that are 1GB+ ... what happened!?

PS. This distribution sure does sound like Beowulf

Excerpt:

"The project is exploring a replicated-kernel OS model for the Linux operating system. In this model, multiple Linux kernel instances running on multiple nodes collaborate each other to provide applications with a single-image operating system over the nodes. The kernels transparently provide a consistent memory view across the machine boundary, so threads in a process can be spread across the nodes without an explicit declaration of memory regions to share nor accessing through a custom memory APIs. The nodes are connected through a modern low-latency interconnect, and each of them might be based on different ISA and/or hardware configuration."

Intuitively, I would think that the way to do this would be as follows:

First, we have the memory hierarchy, disk at the bottom, increasingly faster (but smaller) regions of memory closer to the CPU.

Now, if let's say a process shares memory across nodes, then what seems to make sense would be moving chunks of that memory, as needed, as "cache blocks" to the local machine that needs it.

Observation: That pattern is really not very different than what OS's do with swap files / virtual memory, except that the "swap file" is actually a larger memory region located on a remote node, across the network...

Of course there may be other issues with respect to locking, synchronization objects in memory, thread scheduling, etc., etc., but if you got the above memory access pattern right, then you'd go a long way to enabling such an OS...

Can you please stop putting "Excerpt" in your comments like that? It's enough to use quotation marks or ">" or some other visual marker to show what you're quoting.

The trouble with the way you're doing it is that it signals that the entire comment is an excerpt, and since you've done it a bunch of times, that makes your account look like a single-purpose account or a bot, neither of which are allowed here. I was going to post a totally different reply asking you not to do those things, and then I noticed that actually you're posting normal replies with a nonstandard header.

Hi Dan,

I would like to profusely apologize for using "Excerpt" in my comments for places I excerpt other content, although, I am hoping you'd understand that the reason I do that is for my own way of organizing my own notes... Yes, I understand that HN is a public forum, so in order to please the forum rules and regulations, so from this point forward I will try hard not use "excerpt". Even though I will try not to, if I do it in the future, then understand it was an old habit ingrained upon my stiumulus-response cortex, but nonetheless, I will make a concerted effort not to.

Hi Peter,

You're an awesome HN user. Don't sweat it for one second. Thanks!

If you simply put "> " in front of what you're quoting, it signifies the same thing. Or you can put it in double quotes, or put asterisks around it and it will get italicized.

Will do!

Conversely, you are an awesome HN Admin! Keep up the great work with HN! <g>

Very interesting project, thanks for sharing!

The "open positions" page might interest some people here. They have two 2+ years postdoc positions open with no teaching duties at Virginia Tech.

http://popcornlinux.org/index.php/positions

Requirements are a bit overloaded though..

Whoa! I interviewed to work on this project back in my undergrad days. Was wondering what happened to this.

I note some comments saying that this is more for HPC. I’ll point out that most mobile ARMs have different kinds of cores on the same die (the so-called big.LITTLE approach), and of course often have utterly different functional units (GPU) as well.

From the name, I expected this to be a distro for a TV set-top Box to watch movies on.

Seems to be related to ChronOS Linux at https://www.ssrg.ece.vt.edu

Linux Clustering OS

So joomla + http is still a thing in 2020....

Why would it not? It's still very convenient to use a CMS like Joomla for non tech people.

Http is a shame of course, nowadays most web hosting providers do it automatically.

Joomla has never been great compared to its competitors. It's been many years since I've even heard the name. The are tons of better cms platforms no a days

Made my first money as web developer in ~2006 with joomla actually

Kinda like Lustre?

Yes, kinda.. but lustre is for data, whereas popcorn is for computing..

This seems to be old and possibly abandoned, given that:

- The website is served using a self signed cert for a different domain created in 2011 (!)

- The last download is from March 2013 for linux 3.x

- The last commit to the git repo was in 2016

- The last post on the mailing list is an unanswered question from 2015

I would be interested to hear if anybody on HN knows of a similar project that is being maintained.

The project seems still alive and active, at least according to its recent RFC'ing from LKML [0].

For some more information, Phoronix also recently wrote an article [1] about it, also the GitHub repo [2] seems to be somewhat active (last commit was 06-02-2020).

[0] https://lore.kernel.org/lkml/cover.1588127445.git.javier.mal...

[1] https://www.phoronix.com/scan.php?page=news_item&px=Popcorn-...

[2] https://github.com/ssrg-vt/popcorn-kernel

Ah, you're right, it does appear to be still alive. If anyone from the popcorn linux team sees this, please consider updating the outdated portions of your website (and getting an HTTPS certificate). Had I never read this comment I would have gone on unaware that the project was not dead.

The homepage has a "News" section front and center with articles created as recently as March of this year.

The homepage (the submission URL as I write, but perhaps you saw something different?) leads with News:

  - March 2020 ...
  - [x3]
  - [big font] October 2019 ...

I think it's kicking.

from https://github.com/ssrg-vt/popcorn-kernel/branches

upstream-rfc Updated 5 days ago by ahughes12

upstream Updated 19 days ago by jnarf

nginx_heterogenous_fixes Updated last month by AshwinKrishn