They initially ran a version of the same "CastOS" (we didn't call it that then) the same as what ran on Chromecast sticks. But the landing screen on it is richer than what boots on a Chromecast, and it had an interactive gesture based touch screen. CastOS is just a Linux distrib with some boot and build stuff stolen from Android, but the user space for it is basically booting straight into a chromium based web shell.
Before it (the display assistant) launched, I wrote (and ported from ChromeOS) some of the gesture stuff on it. And some of the accessibility features (magnification).
The "Dragonglass" UI that ran on it was originally built as a HTML/TypeScript stack application. Later it was rewritten in Flutter/Dart, ran for a while like that on CastOS using a Wayland-based wedge layer; and now it runs on Fuchsia.
(I don't think any of this is particularly internal or secret anymore since it's old news and replaced, and most of it was visible from the public chromium repo. Also they can't fire me anymore because I quit ;-) )
But if you mean UI of Android TV itself, then it seems to be a choice of a suitable SoC, or rather not picking an underpowered one. For example, Nvidia picked well for their Shield line.