The red button in particular is used in the UK for lots of TV services. It's not unusual to hear the phrase 'push the red button for X' on TV.
X is usually alternative video feeds of live/sporting events, or different sporting commentary
You can also use the red button for other information such as news headlines, the weather, sport scores. Grandpa uses this, youngsters have their phones and the internet
On that note, I somehow recall seeing TVs shipped with 2 remotes - the 'full one' and a simple one like you describe, with the expectation that people will choose which they prefer, and members of the same family might have opposite preferences.
So really, the Apple TV remote isn't that extreme. It's just a remote for an Apple TV set-top box and not a television.
Then those features don't exist, and that's fine. Is our hypothetical Luddite going to be switching audio streams to SAP? Playing with the picture-in-picture feature? Is he going to know what TTX/MIX means? DMA? E.MODE?
No. Undocumented features are also non-existent features. And for most users, the instruction-manual doesn't count.
It's a TV.
Give basic menu-nav buttons - 4 arrows, OK, Back, show/hide menu.
Give the basic TV tuner buttons - the 0-9 buttons, channel up/down, "Guide" menu button. There's the entirety of your TV facility. Guide needs arrow keys, but our main menu nav already provides that. We don't even really need a "Guide" button, "Guide" can be just be the default view of "Menu".
Volume control, input-selector for its functions as a monitor.
That's it. Notice something? None of those require horrifying abbreviations. We already have standard symbols for all of them. No unlabeled colored buttons. Most of those buttons don't even require words because they're so common we have symbols for them. We just covered all of Grandpa's uses - he wants to change channels and change the volume. We're done. Everything else? Your crazy abbreviated buttons are even more user-hostile than the menu, because at least entries in the menu have vowels.
You could easily cut 6 rows off the Samsung I used as an example and Grandpa would still be happy. Happier even because he's no longer confused about all these crazy weird tiny illegible buttons on his remote. You could even make a large-print version for him and it would be smaller than a dinner-plate.