One problem I've found specific to the Pi (and likely most of the other lower powered ARM chips out there, from what I've seen from Rock Pi, Pine boards, etc.) is that the PCIe layer is not 100% supported on the hardware level as it should be if you want to support features like IO bar space (nonexistent on ARM).

On top of that, until the past couple years, there were only a few exotic ARM builds (probably mostly in custom or enterprise land) that even had standard PCIe slots, so many vendors didn't (and still don't) test their drivers—even if compiled for ARM—on any real hardware.

I hope this changes, especially since Macs now have ARM64, the Pi CM4 supports it, and some other inexpensive SBCs have 4x or even 8x lanes on the more expensive ones (is there an ARM motherboard out there with a 16x lane bus?).

For the Pi in particular, until the CM4 came out, very few people would ever get access to the PCIe bus on the regular Pi 4 model B, so only enough work was done on the kernel side to ensure the VL805 USB controller worked, and a few bits were just never fully implemented or tested (e.g. BAR allocations, MSI-X support, some memory access functions).

Have you talked to broadcom about the PCIe IO space problem? Most other RISCs handle IO space by just mapping into an MMIO space instead. PowerPC, MIPS, and other ARM SoCs have been handling this for decades at this point. It feels weird that their root complex would choke on that, and more likely that there's some unknown config knock you have to setup that hasn't been documented properly. I feel like you might have enough of a publicity reach to get an actual answer on this one.

Quite some recent controllers outright don't implement it, and newer devices don't require it...

Newer GPUs still require it.

> there an ARM motherboard out there with a 16x lane bus?

For servers at sky-high prices only.

For $699, your closest choice might be the NVIDIA Jetson AGX Xavier... PCIe 4.0 but with a x8 slot, and an NVMe M.2 x4 PCIe 4.0 slot too.

SolidRun HoneyComb has about the same slots and price, but instead of weird nvidia closedness you get first-class upstream EDK2 support, everything is about plug & play with ACPI.

The HoneyComb is at PCIe 3.0. Sadly it's at ARMv8.0 (w/ Cortex-A72) so that removes it from consideration totally for me.

And the Xavier has an official-but-still-beta EDK2 firmware available too, which includes ACPI support and can even boot Windows. (https://developer.nvidia.com/embedded/downloads#?search=UEFI...)

Yes, GPU 3D acceleration is currently not supported there yet, but the LX2K doesn't have a GPU at all anyway...

You mentioned work on the kernel side. Is this entirely a hardware issue or are the Linux pcie drivers for the Pi also lacking functionality?

It's a mix, and from my work and discussions on getting a GPU running (or not...), it seems to make debugging even harder if I can't figure out if it's bad memory access or an actual hardware feature that's just not there.

I've been tracking progress on this site / repo: https://pipci.jeffgeerling.com/

You can, with properly written drivers and an open OS.

GNU/Linux obviously is amd64 centric. There's still a tremendous amount of dismissiveness towards anything that's not amd64 (we're even seeing it towards 32 bit x86). So if something doesn't work properly on a non-amd64 system, it's usually not considered a high priority.

The BSDs are much better about this. If something doesn't work, that means it's broken even if it works where 98% of people use it, so it gets fixed. I can, and have, plugged LSI and other RAID cards in to several different non-x86, non-amd64 systems and had them work exactly as expected.

Yeah, I partially wondered because NetBSD specifically has docs talking about ex. plugging "PC" cards into a "Mac" motherboard and being quite emphatic that it should always work because drivers were portable and orthogonal. Sad if Linux isn't doing as well:\

I would guess the peripherals of the processor, bugs in the hardware and firmware etc.

That and the drivers are still murky blobs that I would guess on average don't have much thought put into them from a pure software engineering perspective