Since the GPU starts before the CPU, you need a blob to boot the GPU before you can start open code on the CPU. There is open GPU boot firmware being worked on but it isn't ready yet and relies on GCC support for the GPU, which isn't mainlined yet and the authors don't intend to mainline. So probably RPi will need blobs to boot for the foreseeable future.

https://github.com/librerpi/rpi-open-firmware https://github.com/itszor/vc4-toolchain/issues/7

As far as I know the efforts in this direction only worked for Raspberry Pi 1, not the subsequent ones - or are there some news for later devices too?

I’ve played with this recently in setting up a shared netboot server with Debian for my pile of ARM SBCs (I really should write up that blog post...). The upstream kernel’s device trees are different from the ones supplied by the Raspberry Pi Foundation, with some slightly different names for devices, and basically no easy way to use overlays. Other than that minor obstacle, it’s perfectly doable to build up your own image with debootstrap and chroot with 100% Debian repositories (or go the easy way and download one of the prebuilt images [0]). You do still need the closed-source blobs (bootcode.bin, start.elf, fixup.dat) for boot though, which are made available in the non-free repo.

The 4’s hardware support is still being merged upstream last I heard though, so the easy way for that is still to use the Foundation’s kernel (which is what Ubuntu does, probably to also make overlays work as expected).

0: https://raspi.debian.net/

You only need boot blobs. Accessing the GPU, no. VC4 driver for the Pi3 and older is there, and V3D/VC5 for Pi4 is the only driver, AFAIK there is no proprietary driver for the Pi 4 GPU at all, only the open one.