It uses the Arduino GSM shield, which itself uses the M10 by Quectel, which I believe is a proprietary design.

I know they call it hardware because it's hard, but is it possible to do maybe GSM/Edge with a common SDR, disregarding license and legalities?

Fabian Bellard built an LTE basestation using SDR (so almost everything is done in software). So a GSM mobile stations should be possible, as it has less capabilities.

http://www.bellard.org/lte/

Again, as you said, legalitites. A DIY device would be difficult to approve by whatever telecom supervising agency your country has. That's one of the reasons the current ecosystem of blackbox GSM modems (such as the one used by this Arduino phone) exist. These modems run closed-source software that can theoretically react to network events, possibly get updated OTA. So there's the icky part from a security viewpoint.

SDR -> Software Defined Radio -> receiver. You'll need a transmitter too.

Though I'm sure some clever person somewhere hacked one to do just that.

SDR transceivers have been on the market for quite some time; have a look at the HackRF, for instance.

At least Wikipedia says a software defined radio can also include a transmitter. I've seen a couple programmable transceivers, but virtually all of the cheap SDRs are just receivers (usually repurposed TV sticks).

I know that someone at least build a setup where they could record and decode GSM with a reciever, but I can't find it right now.

...and the Quectel M10 itself uses the Mediatek MT6223, a common very-low-end feature phone SoC.

I'm not aware of any open basebands existing, and I'm not sure how they'd pass regulatory requirements.

There is an open GSM base station system though, OpenBTS. But you definitely can only use that for testing.

The FreeCalypso Project is working on a libre phone. Looks like they're using a baseband processor from a TI Chipset.

https://www.freecalypso.org/