Nothing about the Raspberry Pi’s video system is normal.

Basically, disregard everything you’ve learned about KMS from the world of Raspberry Pi. It’s only relevant to the quirkiness of the Raspberry Pi’s half-closed ecosystem.

The F in fkms stands for “fake” because they created a quirky abstraction to fake KMS compatibility. The non-fake driver has been all sorts of problematic but it’s getting better.

But none of this Raspberry Pi specific weirdness should be extrapolated to the KMS/DRM subsystem in general.

> Nothing about the Raspberry Pi’s video system is normal.

Nothing about the Raspberry Pi at all is "normal," and it's a very useful filter for people who have read about the various deep weeds of firmware/OS development and those who have actually gotten their hands dirty on a Pi - because the first group will express admiration about the "open and well documented hardware," and the second group will laugh at the first group and start cursing the various non-and-poorly-documented nonsense that's in the Pis.

They're nice devices, and work well enough, but they are architecturally insane and unlike anything else except the previous Raspberry Pis that were hacked to to make the new ones.

The Pi1/2/3 are more or less an (undocumented) GPU with an ARM core bolted on the side. The GPU boots the system, allows the ARM core some access to the DRAM (not at the same addresses as the GPU sees things, so be sure which version of memory addresses you're looking at), and most of the hardware control is done by asking the GPU to please do this thing for you through the mailbox interfaces. In the same way that userland applications make syscalls into the kernel, the kernel running on a Pi makes syscall-like requests into the GPU to do just about anything.

The reason you see things like "All the interrupts are only ever on a single core" is because the interrupt controllers simply don't allow anything else. The Pi2 BCM2836 interrupt controller feels a little bit like an intern's project, using the BCM2835 bolted on the front as an input for the rest of the hardware. At least the Pi4 finally gets a GIC...

They're fine, sort of... as long as you're OK with the only documentation for some of the hardware being "This Linux driver works with it" and "Well, people have reverse engineered most of it." But they're absolutely insane, architecturally, they're poorly documented, and this doesn't seem widely known.

Doesn’t sound too insane; sounds like an architecture designed for real-time professing, where you’ve got an “IO processor” that boots the system and handles all the interrupt-heavy work, and an “application processor” that can be solely dedicated to your workload and will only be interrupted by the IO processor if it explicitly asks to be by first making a request to it.

Very similar to previous-gen game consoles, e.g. the Wii U’s “Broadway”

Does the pi allow to put own routines straight onto it's "IO processor"?

Hopefully the work done by the LibreRPi folks helps things get a little less weird:

https://github.com/librerpi/ https://gwolf.org/2022/04/how-is-the-free-firmware-for-the-r...

I'm quite familiar with Clever's work. I understand how it's put together at a pretty deep level - and it's still weird. It's not the slightest bit like anything else that is used as a general purpose computer in the layout of "how pieces talk to each other or how they interact with hardware." You can draw some parallels to the ARM Trusted Firmware interfaces if you squint hard enough, but it's still a really weird hardware layout for what it's commonly used for.

"Non- and poorly-documented nonsense" is exceedingly common throughout the embedded hardware ecosystem. The Pi is doing well on a relative scale of freedom and openness, not an absolute one.

> The Pi1/2/3 are more or less an (undocumented) GPU with an ARM core bolted on the side.

Is the Pi4 any different/better?

I don't know the Pi4 architecture deeply enough to offer an opinion on it, sorry. Ask me in a year or two and I'll likely know it a lot better.

Well, they fucked up their implementation of USB-PD with a rookie-level mistake, it took ages for them to support booting off USB, the 5V rail is noisy...

Raspberry Pis were initially just a way for the Pi foundation to help Broadcom sell SoCs nobody wants anymore but they still had lots of stock/tooling for. Sort of like "hybrid" bicycles are how bike manufacturers milk more money out of hardtail (non-suspension) MTB frames.

It's almost the same overall architecture, just a lot of pieces have been upgraded to be wider and faster, and some peripherals added.

Are Broadcom's ARM chips that half-baked in general, or are these quirks specific to the ones used in the Raspberry Pi?

Ah, thanks for that.

> Basically, disregard everything you’ve learned about KMS from the world of Raspberry Pi. It’s only relevant to the quirkiness of the Raspberry Pi’s half-closed ecosystem.

It's funny, I just posted a comment about being afraid to dig into KMS code for fear a wizard would show up and tell me I'm looking in a wrong or irrelevant place. Luckily, your comment preempts that process.

And also luckily, I've only thrown flags at Chromium provided by other wizards so far. So my ignorance about KMS remains pure. :)

> The non-fake driver has been all sorts of problematic but it’s getting better.

So... are there like two different teams, one Pi-specific 'f', one general 'not f', racing to see whose module will become stable first?

Do both teams stream on Twitch? If not they should.

The dtoverlay parameter (device tree overlay) allows manually inserting device nodes into the device tree loaded by the system.

vc4-fkms-v3d is a module, vc4-kms-v3d is an entirely different module. These correspond to files somewhere on your device that describe the device and how to operate it.

The -f- version seems to do more video processing in the firmware rather than in the kernel. I can't tell you exactly what they do differently, but I'd assume that the -f- variant uses some kind of Broadcom magic and the non-f-version runs mostly in-kernel and therefore has different side effects when it comes to video processing.

Switching between these changes the way the kernel deals with graphics, so that's probably why you see the mouse being smoother in one version and hardware acceleration being unreliable in the other.

I can't tell you why Chromium is being weird. It could be a Wayland/X11 issue, it could be a driver support issue, it could be firmware issue, or maybe it's got something to do with sandboxing (Ubuntu and its Snap sandboxing have caused hardware access issues for me in the past). I have no idea what DRDC is even supposed to do, I'm guessing it's an optimisation in the Chrome rendering engine but who knows at this point.

To get the answers you seek, you'll have to dig into source code, I'm afraid. Perhaps if you reserve engineer the requirements for Chrome to enable the setting you'll find a hint at what you need to do to your rendering software to get it to support Chrome's code.

> The -f- version seems to do more video processing in the firmware rather than in the kernel. I can't tell you exactly what they do differently, but I'd assume that the -f- variant uses some kind of Broadcom magic and the non-f-version runs mostly in-kernel and therefore has different side effects when it comes to video processing.

What does it mean to do video processing "in the kernel?" I assume that cannot mean software decoding-- if that were the case I wouldn't be able to play back smoothly.

This is only about "kernel mode setting" - basically interfacing the hardware that drives your displays, sets up HDMI (if you use that) and so on. For the Raspberry Pi specifically, the "fkms" for fake KMS means that you use the proprietary RPi firmware to talk to the display and setup your screen, while the kernel just throws over a framebuffer. If you use the non-fake KMS, you have the kernel do all that lower level setup but you get more direct access to the hardware that allows you to use multiple planes and so on.

Why have both? The firmware, being derived from what was a set-top box, has a lot of proprietary magic to make things work with displays, also specifically with things like the official RPi display.

I can tell you why this is happening! I think!

OK, so first off, FKMS vs KMS.

KMS is Kernel Mode Setting. This is the kernel's framework for setting video modes, abstracting over HDMI/DSI initialization, timing parameters, etc. etc. across platforms.

FKMS is still KMS, but it's a special KMS driver for the Pi which sets up HDMI using the DispmanX API into the Broadcom proprietary OS that runs alongside Linux on the Raspberry Pi. This is nice because it's dead simple for Linux (Linux quite literally gets to say "yo, gimme a 1920x1080 layer!") but bad because it relies on a proprietary blob and the blob can't be extended.

Native KMS, on the other hand, sets up HDMI using native device access to the actual registers which affect the HDMI encoder, DSI link, and so on. This is more extensible and isn't closed source, but it's also mega complicated.

Now, here's the kicker on your Pi. Other commenters are correct that normally, KMS should just affect mode setting, not rendering. But, on the Pi, this isn't true. KMS and FKMS _also_ affect the way _layers_ are configured in the DRM (Direct Render Manager) subsystem.

Why? DispmanX isn't just a mode setter, it's also a scaling and compositing (layering) engine!

So, in the course of bypassing DispmanX, the DRM system in KMS mode now needs to act differently - it needs to configure 2D compositing planes through the KMS system instead of through DispmanX.

What's on a separate 2D compositing layer?

The legacy cursor.

Here's the switch where RealKMS is responsible for setting up the cursor compositing layers, a task which in FakeKMS was already handled by DispmanX:

https://github.com/raspberrypi/linux/blob/aeaa2460db088fb2c9...

https://github.com/raspberrypi/linux/blob/1fe617a917eac75800...

Now, _why_ this makes your cursor lag, I don't know, but that's the difference.

> Other commenters are correct that normally, KMS should just affect mode setting, not rendering.

Clearly those commenters don't have much experience working close to the wire around GPUs then. Scanout used to be very closely tied to rendering, and in some ways it still is. You can't take an arbitrary texture in GPU memory and scan it out to the display.

Windows has this thing called VidPN to work around the dependencies between source and target modes which gets even more complicated with VRR and virtual resolutions (retina).

https://docs.microsoft.com/en-us/windows-hardware/drivers/di...

Hardware composer is the Android version of this: https://source.android.com/devices/graphics/implement-hwc

This has become a lot more important with smartphones than desktop systems. You now have often very different silicon for things like video decoding, graphics and scanout - and to achieve reasonable battery time you need common use cases like watching a video to not involve the GPU at all.

Dedicated video decoding silicon is not part of the GPU?

Video decoders are usually fixed function blocks, separate from the SMs (which I believe the parent is referring to by "GPU") but are still usually on the same physical silicon as the GPU (in case of SoCs, also shared with CPU).

They're exposed as an independent engine on the GPU/SoC and can operate with most of the chip power gated.

You're toying with forces that even the most learned KMS scholars haven't fully investigated. Remember: the kernel, and by extension KMS, evolved to it's current state through the collective intellect of many humans. No one knows what's in there, and those who get close are driven mad, like a modern-day kabbalah. It's best not to think too much about it, but you should be safe trying random incantations you find in books of arcana and witchcraft.

I'm almost afraid to try to peruse code at this point. I fear I'll be on the verge of understanding some relationship between this castle full of abstractions to my video stuttering, then find some wizard in a corner who will tell me that's old and the real codebase is in another castle.

If you don't want to experience this type of nonsense, use hardware from vendors that make good-faith efforts to fully support their hardware in mainline kernels, like Intel or AMD's hardware. Just treat graphics and acceleration on the RPi as a neat trick.

You can pick up SBC-like computers with Intel integrated graphics and not have to deal with these problems from vendors' unsupported hardware, and there are some with AMD APUs. There are also SBCs that have PCIe slots that you can stick your own graphics cards into, but I'd stick with x86 machines because embedded ARM devices almost always require forked kernels that might not support everything you'd want them to, including your otherwise-supported graphics cards.

Some Mali graphics chips also work pretty well with Panfrost/Lima.

Do you have any experience with Pine64's creations and whether they involve this fuckey?

I don't, unfortunately. It looks like their products use Mali 400 MP2 GPUs, which seems to have support via Lima. I had several boards with Mali-T628's, and while they work, I ran into headaches a lot a 6+ years ago, like issues with GPU accelerated video, browsing in Chromium and desktop compositing. I've heard from developers that work on those boards that the GPU support is much better than it was a half decade ago, though.

Personally, I won't be buying more ARM SoCs unless they start supporting SBSA and have mainline kernel support for all of their peripherals. As it stands, the only hardware that I'm aware of that has full mainline support for everything, including GPUs, are boards with Intel iGPUs or AMD APUs.

KMS only means Kernel Mode Setting, it's a way for the kernel to handle display resolutions so it can set the proper resolution for a display during bootup and enables instant TTY switching.

DRM is the direct rendering manager, it's mainly an interface for applications like X11 to talk to the GPU directly (ie 3D Acceleration and Video Decoding).

If KMS is causing things to be slow, it might be because the video driver is not working and the software framebuffer is being used. At higher resolutions that'll be slow.

If Chrome doesn't have the option to enable DRDC then the driver used likely doesn't support DRM or the window manager is not passing it properly.

> If KMS is causing things to be slow, it might be because the video driver is not working and the software framebuffer is being used. At higher resolutions that'll be slow.

But then how could using a software framebuffer on an underpowered thing like the RPI400 cause the video to play smoothly?

>an underpowered thing

I would not call Raspberry Pi 400 underpowered.

The Cortex-A72 cores are fairly fast.

I know from first hand experience that cpu struggles to play MPEG-4 video in software.

What codec from MPEG-4, and what software implementation?

DrDc is unrelated to the issue here. As another commenter guessed, it's just an optimization to Chromium's graphics pipeline. Chrome has a number of processes that need to execute GPU work. For example, there's the renderer process which actually draws the page content into tiles (we refer to this as "rasterization"). There's at least one renderer process per page because of Site Isolation, but there can be more like in the case of iframes. There's also the Viz process that (among other things) handles the composition of all of the tiles generated by the renderers into the final framebuffer (this is known as the Display Compositor). Each of these processes funnels their GL calls into a separate thread called the GPU main thread which as part of Chromium's security model, is the only thread privileged to interact with the actual system graphics drivers.

The insight behind DrDc is that display compositing is higher priority than rasterization, so having both share work on the GPU main thread is bad for performance. DrDc is a large effort, but at the risk of oversimplifying, it's main purpose is to create a new GPU thread that the display compositor can use exclusively. DrDc is still rolling out on most platforms which may explain why it's disabled on your device, but regardless, it should not help with the video playback issues you're seeing.

I'm a little surprised people expect Chromium to run decently on the Raspberry Pi.

I was surprised, too. But it did work on Raspbian buster minus one.

Then I upgraded to Raspbian buster to get a more recent version of Chromium since HBO Max was blocking me based on whatever version shipped with (buster minus one).

At that point I started shooting various startup flags at Chromium from the web of wizards' incantations until video started to play again. That including removing the 'f' per the incantation I mentioned. This cured video playback health but it put my mouse under some kind of latency spell.

I think I could beat this level with a drowsy mouse if only I could find increased magic for the subtitles somewhere.

Does anyone know if there is a Chromium arg I can use to just farm some magic?

Laggy mouse is the surprising part I think. It used to be that you could count on the mouse still moving smoothly even when the rest of the system was falling to pieces. I understand this was because the mouse was a hardware-rendered sprite controlled by hardware interrupts. I don't claim to understand the details, but my impression is things have changed a lot.

The mouse lags. Sometimes it even turns on an invisibility cloak which I didn't even know was possible.

I need to see if there are any RPI400 Chromium Video Acceleration Challenge players streaming on Twitch, maybe they'd have some clues. There are bound to be a bunch of useful weapons to beat this level but I don't know how to unlock them.

Like someone else mentioned it's probably because the video is in some kind of pure framebuffer mode.