I am always interested to hear when a language is ported to a new plaform, in this case Python (or a subset of) on GRUB. The question would be why? Is this because you have lots of test cases already in Python and can easily deployed those test cases on using this?

Originally, it was because we had a lot of one-off test cases not written in Python. Before BITS, most BIOS test programs consisted of one-off DOS programs, or later one-off EFI programs. If you wanted a new test, or you wanted a customer to quickly gather some information for you, you sent them a new test program, or a bootable image. Now, many of those one-off tests have become Python one-liners, and you can use Python to explore the system with an interactive REPL.

Python also lets us provide additional APIs to make scripting easy. For instance, we include the complete ACPICA interpreter, so you can evaluate an arbitrary ACPI method with arguments and process the result. And we have an FFI with EFI support, so you can locate and call an arbitrary EFI protocol.

Wow. I thought this was one of those "just because you can," projects, but it looks like there are some real practical uses for firmware developers/testers.

Yeah, we wrote BITS almost entirely for practical reasons. Several groups at Intel use it to test both the BIOS and the CPU itself. (You have to have enough functionality to boot an OS, but once you have that, BITS can easily test things like CPU power management or MSR configuration.)

The module that displays presentation slides using EFI_GRAPHICS_OUTPUT_PROTOCOL, on the other hand, was definitely "because we could". https://www.youtube.com/watch?v=bYQ_lq5dcvM#t=22m56s

Thanks for the detail answer.

Does your work have a high dependencies on Intel CPUs? ie, have your heard of it being used on other CPU architecture, eg ARM or PowerPC?

It could be ported to ARM systems running EFI with some effort, if someone wanted to do so. It wouldn't make much sense to port to a completely different BIOS architecture such as those on PowerPC, though.

Did you modify the CPython binary to run in the BIOS, or did you write a new interpreter from scratch? What was the hardest part of implementing this?

We build CPython's source into BITS, which runs as an EFI binary. (Or as a non-EFI image bootable on a BIOS, but that version doesn't include network support.) We have almost no changes to CPython itself, other than a couple of minor fixes to enable some additional calling conventions in the ctypes module.

The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which required some very fiddly assembly. That libffi support allows us to call EFI protocols from Python without writing any C code.

The hardest part of implementing networking was handling the impedance mismatch between POSIX select and EFI asynchronous calls. POSIX select lets you ask "do any of these sockets have data to read", without actually reading the data; likewise for pending client connections to accept(). EFI expects you to provide a buffer to read the data and a callback for when the data gets read. So, the only way to ask for pending data is to buffer that data for the application to read later. This requires additional copies, limiting performance. (You might have heard of "zero-copy" networking; this is "several-copy" networking.)

It sounds like IOCP on Windows. For example, asyncio.ProactorEventLoop and multiprocessing's PipeConnection can use it.

I apologize in advance for my lack of detailed knowledge about low-level IO, but,

Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard?

I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges pyuv calls directly to EFI without the stdlib as an intermediary?

> Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard?

No, it wouldn't. libuv depends on OS facilities like select, poll, or epoll, and then builds a high-performance event loop on top of them. We needed to implement those facilities themselves.

In theory, if we didn't care about Python's standard library modules, we could potentially port libuv to support the asynchronous callback mechanism used by EFI. However, we also wanted to write as much of this code as possible in Python, rather than C, to make it easier to maintain. That's why, for instance, we didn't implement the Berkeley sockets API in C to use the CPython socketmodule.c, but instead implemented our own _socket.py in Python directly on top of the efi module.

How big is the base image, e.g. printing "hello world" to the console? Is it modular, e.g. if you don't need networking[1] how hard is it to not include networking?

[1] I'm using networking as an example package, not knowing what other semi-optional packages could be used as an example.

> How big is the base image, e.g. printing "hello world" to the console?

46MB. However, that image includes three separate complete copies of all the compiled code (4.5-6MB each), so that it simultaneously boots on 32-bit BIOS, 32-bit EFI, and 64-bit EFI. It also includes the complete source code (24MB) needed to reproduce the image, to make it self-contained and redistributable without needing to have a separate distribution of source. And it includes .py files for all the BITS .pyc files (other than the Python standard library), to simplify debugging and hacking. A quick check suggests that you could pretty easily build an 8.4MB image, if you cared about the size.

> Is it modular, e.g. if you don't need networking[1] how hard is it to not include networking?

No. We haven't needed a smaller image, so we haven't worked on that. Typically, we boot it from a USB drive or hard drive, so storage space isn't a concern.

Networking, though, consists almost exclusively of Python code, and the compiled .pyc files for that don't take up much space; not a good target to start with if you wanted to reduce size.

Awesome stuff! What are your future plans in this effort?

Could you envision a (G)UI toolkit written in Python that could be used to write applications that run in EFI?

> Awesome stuff! What are your future plans in this effort?

On a small scale, moving as much as possible from C to Python using ctypes. Much of the C code pre-dates us implementing ctypes support, so it provides bindings to things like ACPI via the Python C API. We'd like to re-write that using ctypes, which will result in far less C code and no manual management of things like reference counts of Python objects.

On a larger scale, we're outgrowing GRUB's menu system. It's been incredibly helpful to get started, but we'd like a more usable and flexible UI than GRUB menus based on dynamically generated GRUB cfg files. (Several of the menus consist of lines like "source (python)/test.cfg", where "(python)" is an in-memory GRUB filesystem implemented in Python, and Python code enumerates the test hierarchy at boot time to generate test.cfg.) In particular, we'd like to be able to have simple UI elements to select one of a set of configuration options and display the current value directly in the menu.

So yes, I can envision a UI toolkit written in Python. :)