News for hackers. Stuff that matters.
Re:Actually Portable Executables by abhinav22 on Sunday February 28, 2021 @05:59AM (#26292444)
Can somebody explain in layman’s terms what this is all about? I tried my best reading the other thread, but it went well above me.
Re:Actually Portable Executables by Gaelan on Sunday February 28, 2021 @06:12AM (#26292508)
Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things:

1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format.

2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.

Re:Actually Portable Executables by abhinav22 on Sunday February 28, 2021 @06:38AM (#26292648)
Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!
Re:Actually Portable Executables by jart on Sunday February 28, 2021 @06:58AM (#26292757)
That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
Re:Actually Portable Executables by fctorial on Sunday February 28, 2021 @07:25AM (#26292901)
Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.
Re:Actually Portable Executables by jart on Sunday February 28, 2021 @08:03AM (#26293128)
The only programming environment I used to build this was Emacs on Linux. The rest was simply researching the magic numbers needed to use the binary interface of each system. I figured life's too short to be pulling my hair out with MSVC/Xcode warnings.
Re:Actually Portable Executables by fctorial on Sunday February 28, 2021 @12:51PM (#26295250)
s/programming environments/systems/
Re:Actually Portable Executables by lambda_obrien on Sunday February 28, 2021 @10:29AM (#26294018)
Does cosmolibc allow you to run on bare metal? If so, I've got an old project in mind that you may have just made happen, and I'm very very excited to know!
Re:Actually Portable Executables by collinmanderson on Monday March 01, 2021 @02:14PM (#26306882)
"Actually Portable Executable"s can boot from BIOS if that's what you're wondering.

https://justine.lol/ape.html

Re:Actually Portable Executables by lambda_obrien on Monday March 01, 2021 @05:49PM (#26309566)
Cool, thanks! I missed that before. It makes a project idea I had way more realistic for me to implement.
Re:Actually Portable Executables by thysultan on Sunday February 28, 2021 @07:33AM (#26292935)
How does one unzip the .com file? Is that even possible or have i misunderstood how it works?
Re:Actually Portable Executables by jart on Sunday February 28, 2021 @07:50AM (#26293041)
I recommend using InfoZIP on the Linux or Mac command line. For example, to list the files that are in your .com executables you can run a command like:

    $ wget https://justine.lol/redbean/redbean-2021-02-27.com
    $ unzip -vl redbean-2021-02-27.com
    Archive:  redbean-2021-02-27.com
     Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
    --------  ------  ------- ---- ---------- ----- --------  ----
         426  Defl:N      215  50% 02-09-2021 09:24 f9cc9464  tool/net/redbean.css
         896  Defl:N      509  43% 02-09-2021 09:24 eb74f8a0  tool/net/redbean.html
       16958  Defl:N     6093  64% 02-09-2021 09:24 6a76bd39  tool/net/redbean.ico
        5073  Stored     5073   0% 02-09-2021 09:24 86c2afe6  tool/net/redbean.png
         554  Defl:N      285  49% 02-09-2021 09:24 0aa6870a  usr/share/zoneinfo/Beijing
        2335  Defl:N     1093  53% 02-09-2021 09:24 6ccd4450  usr/share/zoneinfo/Berlin
        2453  Defl:N     1170  52% 02-09-2021 09:24 7fed6d80  usr/share/zoneinfo/Boulder
    ...
Another ZIP tool that works well with APE is Windows 10 file explorer. But you have to change the extension from .com to .zip temporarily.
Re:Actually Portable Executables by A12-B on Sunday February 28, 2021 @10:07AM (#26293869)
What happens when intel goes bust and we all have to switch to new processors
Re:Actually Portable Executables by OOPMan on Sunday February 28, 2021 @12:44PM (#26295183)
A little optimistic about Apples SoCs aren't you?

Also...given that AMD exists and the x86 instruction set is not some magic potion ingredients...

Re:Actually Portable Executables by shrx on Sunday February 28, 2021 @07:04AM (#26292790)
They also run on AMD ;)
Re:Actually Portable Executables by munawwar on Sunday February 28, 2021 @02:48PM (#26296407)
So it's like a compatibly layer like wine except built into the binary. Does this bloat the binary as well?
Re:Actually Portable Executables by sgtnoodle on Sunday February 28, 2021 @10:27PM (#26299397)
The only code that really needs to be duplicated would be the lowest level of userspace code that actually calls into the various operating systems. Things like the guts of "open", "read", "write", etc. Regardless of any particular OS's flavor, they're all going to roughly take in the same arguments and return the same results via some flavor of system call.

Each individual function is probably on the order of a few dozen machine instructions. Let's say there's 30 such functions that are implemented and 7 OSes to support. 36307 is about 7500 instructions. The average instruction length on x86-64 is 2-3 bytes. So maybe it ends up around 20KB of code space for a program that uses every possible operating system feature that's implemented?

I haven't looked at the implementation. I wonder how it selects between OS implementations. Does it switch on every system call, use a function pointer table, or do some sort of clever in-memory code rewriting?

Re:Actually Portable Executables by galfarragem on Sunday February 28, 2021 @07:35AM (#26292954)
Will this rush an optimized version of Electron or even deprecate it?
Re:Actually Portable Executables by arendtio on Sunday February 28, 2021 @01:05PM (#26295422)
Well, the main problem with Electron is that every app ships its own browser using tremendous resources at runtime, having huge attack surfaces and lots of unused functionality. So just giving one optimized electron version to all users, certainly does not solve the problem as APE does nothing that changes these problems.

Speaking of deprecating it, it might be one scenario, but developers probably don't just choose Electron because it is platform independent. I mean, there were other frameworks that supported that use-case even before Electron (e.g. QT). However, what is specific to Electron, is that you can use web technologies to build Apps. But that is also nothing you can do with APE.

So as much as I like to see one binary format for many operating systems, I doubt that it will change anything regarding the Electron problem.

There certainly multiple ways of solving it, but in my opinion, Apple and Microsoft must come to a common ground and support some kind of web-view that is built-in to the OS and extended with a platform independent API for the things that normal browsers do not support/allow, like filesystems access.

At some point both supported the Progressive Web App (PWA) movement for a while, but afaik Apple is the one holding it back at the moment. My guess is that they saw their App-Store business at risk. And even if PWAs were a thing, they are still some use-cases they can not cover in the current state (e.g. missing filesystem access).

Re:Actually Portable Executables by mihcsab on Sunday February 28, 2021 @01:29PM (#26295718)
the main problem with electron is js
Re:Actually Portable Executables by Nextgrid on Sunday February 28, 2021 @07:39AM (#26292973)
Technically, I guess you could use this to compile a universal browser binary that would run on all platforms. However, this wouldn't solve any of the problems caused by Electron.
Re:Actually Portable Executables by dj_mc_merlin on Sunday February 28, 2021 @08:55AM (#26293422)
I think his point was that with Cosmopolitan native binaries one would no longer need Electron to provide cross-platform binaries. There is a lack of good GUI frameworks to build this native Electron replacement with though, nor is it certain the effort would not be too high.
Re:Actually Portable Executables by darkfirefly on Sunday February 28, 2021 @07:41AM (#26292986)
Or really any of the problems Electron attempts to solve; the GUI toolkits are still not (generally) not cross-platform in the way Electron is.
Re:Actually Portable Executables by simias on Sunday February 28, 2021 @10:58AM (#26294229)
Note that packaged electron apps are not portable across OS, so it's orthogonal. What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary.

Meanwhile Cosmopolitan doesn't provide a portable GUI toolkit so it can't replace electron on its own. Maybe if you could package Cosmopolitan + Qt or something like that you could end up with something very interesting though.

Re:Actually Portable Executables by cxr on Sunday February 28, 2021 @01:31PM (#26295742)
> What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary.

Likewise for non-GUI apps, like command-line utilities that are implemented with NodeJS. As prior art, Fabrice Bellard's QuickJS is able to compile JS programs to spit out a native binary that requires no external qjs runtime. If you modified it to output Actual Portable Executable binaries instead, it would be even more portable than the programs installed with `npm install`.

I've done some experiments and progressed to writing tooling that approaches that I-want-ridiculous-portability problem differently. It doesn't use APE, so it still requires a separate interpreter (such as NodeJS) capable of executing the compiled output, but the object files it outputs are such that even if you don't have a copy of NodeJS installed on your system (or the right copy of NodeJS installed), then you can piggy back off the interpreter built in to your browser. You get a similar double-click-to-run experience, but when you do double-click to run, it launches in the browser, and you can go from object file to modifiable source trivially, since the object file and the Git repo from which it is built are automorphisms. The downside, if you consider it one, is that it doesn't work with arbitrary NPM modules. You have to really buy in to the scripting language dialect and the development philosophy to get anything out of it.

It would be interesting the use the QuickJS+APE strategy above so that there is _no_ reliance on an external interpreter, at the cost of marrying yourself to x86 and introducing some small hurdles in the path to modification and also giving up the safety of the browser sandbox, since you're back to running arbitrary executables.

Re:Actually Portable Executables by matheusmoreira on Sunday February 28, 2021 @06:18AM (#26292544)
The author somehow came up with an executable file format that's compatible with everything. It's simultaneously a valid shell script, Unix ELF, Windows PE file, boot sector code and zip file.

Then she wrote a standard C library that detects which OS you're using at runtime and branches to the appropriate implementation, allowing the same x86_64 code to run on any supported system.

  if (IsWindows())
      DoWindowsThing();
  else
      DoUnixThing();
Traditionally, these system differences are resolved at build time: only the code for the target platform is included. Why would anyone want Windows-specific code in a Linux build? It's dead code... right? The new run-anywhere executable format makes that code useful.
Re:Actually Portable Executables by arendtio on Sunday February 28, 2021 @01:11PM (#26295504)
> Why would anyone want Windows-specific code in a Linux build? It's dead code...

You might want to ask the thousands of Electron App users how much they care about dead code ;-)

I mean, you are right. But if you offer your users no alternative they might not care enough to choose another product. In addition, some might even value it not having to choose the right binary for their system.

Re:Actually Portable Executables by matheusmoreira on Sunday February 28, 2021 @09:41PM (#26299168)
> In addition, some might even value it not having to choose the right binary for their system.

Yes. The true innovation that made all this possible is the novel Actually Portable Executable (APE) format. This means there's no need to choose: Windows will read it as a valid PE file while Linux, the BSDs and Mac will read it as a valid ELF program.

Having code for all operating systems in the executable has always been possible. It's just that before APE there used to be no point. Windows simply isn't able to load and execute an ELF program even if there is Windows code in it.

ELFs would be incompatible even between Linux, Mac and the BSDs since they would almost always depend on very different dynamic libraries. Cosmopolitan fixes this by providing a standard C library with support for everything. I assume incompatibilities can still be introduced by linking against other libraries.

Re:Actually Portable Executables by IshKebab on Sunday February 28, 2021 @10:07AM (#26293875)
Are you sure? Other answers are suggesting it isn't a valid ELF file and I don't get why it would need to be a valid boot sector or zip file.

You might be right though. The author might be very good at executable hacks but they're a terrible writer. Their explanation explains nothing.

Re:Actually Portable Executables by reikonomusha on Sunday February 28, 2021 @01:53PM (#26295964)
The author seems to be an excellent writer. They just may not have taken the enormous amount of time to answer tons of questions or write tons of documentation people are seeking. One person did all this!
Re:Actually Portable Executables by IshKebab on Sunday February 28, 2021 @03:36PM (#26296826)
You really think so? Here's their description of how it works:

> Here's how it works:

>

> MZqFpD='

> BIOS BOOT SECTOR'

> exec 7<> $(command -v $0)

> printf '\177ELF...LINKER-ENCODED-FREEBSD-HEADER' >&7

> exec "$0" "$@"

> exec qemu-x86_64 "$0" "$@"

> exit 1

> REAL MODE...

> ELF SEGMENTS...

> OPENBSD NOTE...

> NETBSD NOTE...

> MACHO HEADERS...

> CODE AND DATA...

> ZIP DIRECTORY...

Ok well that clears it up. MZqFpD='. Of course!

Re:Actually Portable Executables by matheusmoreira on Sunday February 28, 2021 @10:24PM (#26299385)
Actually I'm not 100% sure. I can barely make sense of it. All I know is it somehow works as if it was a valid ELF. Some dark magic going on here...
Re:Actually Portable Executables by baq on Sunday February 28, 2021 @06:22AM (#26292565)
No worries. You’re in good company. This actually portable executable thing is batshit insane (in the best possible way).
Re:Actually Portable Executables by gostsamo on Sunday February 28, 2021 @06:12AM (#26292510)
In short, there is a "framework" that allows to compile a single binary that can run under both Linux, Windows, bare metal, and many others. Like the Apple universal binaries, but actually universal, not just covering the m1's arm64 and x86 64.
Re:Actually Portable Executables by Gys on Sunday February 28, 2021 @06:16AM (#26292527)
It is x86 only, arm is not supported. But would be a very nice addition :)
Re:Actually Portable Executables by TickleSteve on Sunday February 28, 2021 @06:27AM (#26292589)
non-x86 is emulated and bare-metal limits the portability.
Re:Actually Portable Executables by max_ on Sunday February 28, 2021 @06:17AM (#26292541)
You can compile a program on MacOS. The result, an executable binary, can run on Linux, Mac, Windows, FreeBSD, OpenBSD, NetBSD, BIOS .... the same EXACT binary!
Re:Actually Portable Executables by snarfy on Sunday February 28, 2021 @06:12AM (#26292513)
It makes binaries that can run anywhere by shimming(?) libc functions and values with a custom loader/linker.