It's not so much the decoding and memory as it is that the timing of the HDMI signal waveform at 4K/60fps is a bit too fast to read any portion of with an affordable FPGA. There are FPGAs with specialized DSP hardware that can probably do it, but the ones I know of require pricey IP core licenses and the chips themselves aren't terribly affordable.

That's kinda what I'm asking about here, if there is a something like a specialized chip that can be used to capture single frames from an HDMI stream to act as a buffer, because yeah obviously you don't need most of the data and you only really need to update at a few Hz.

The other option I've thought of is a chip to downsample the signal, as there are plenty of cheap FPGAs or even micros that can handle resolutions below 1080p. Again, I'm pretty sure such chips exist but I don't really know of them because this is just something I am interested in for fun.

A Xilinx one with transceivers should be able to do it though. Yes, not cheap as such, but oh well. Depending on resolution, a 3 or 5GHz transceiver chip from lattice should be reasonably cheap, and IIRCV they have special HDMI chips.