My recommendation is to start with the CPU emulation. The CPU in the Gameboy has decent documentation and there are plenty of implementations to look at if you're stumped. (One I like particular is the core in Higan, a multisystem emulator written in C++.)

I remember seeing some people recommend starting with the NES rather than the Gameboy - any ideas on this?

The main advantage to NES emulation is that it's a very mature area of study. High-quality documentation is much more readily available than for other systems, and there's more public discussion of the specific problems that NES emulator authors encounter (both primarily on nesdev.com [1] [2]). There's even a sort of tool-assisted speedrun of writing an accurate NES emulator, if you're into that sort of thing [3] [4].

[1] https://wiki.nesdev.com

[2] https://forums.nesdev.com/viewforum.php?f=3

[3] https://www.youtube.com/watch?v=y71lli8MS8s

[4] https://www.youtube.com/watch?v=XZWw745wPXY

Gameboy is actually a bit friendlier than the NES to get started with, because of the low number of mappers (programmable chips in the cartridges to do tricks with addressing, NES had much greater need for these than the GB due to the dearth of tile RAM). But Sega Master System is even friendlier, I hear.

Start with Chip-8 or Space Invaders, both of which are substantially easier than the NES and Gameboy.

Took me a minute to realise you were talking about 8080 emulation - not Atari emulation: https://github.com/superzazu/invaders

How did you reach that conclusion?

http://emulator101.com/ is a good resource, as is https://www.reddit.com/r/emudev/.

Or you can just cheat, design a "toy" instruction set, and write something to process it. A couple of years ago I did that, in C/Perl:

* https://github.com/skx/simple.vm

Later I rewrote the interpreter/emulator in go:

* https://github.com/skx/go.vm

The downside with my approach here is that I ignored graphcs, sound, & I/O.

These days I'd recommend you write an emulator for the Chip-8 system..