I was an early adopter. In the beginning, there was actually a handful of us reviewing avatars and assigning ratings. Now it's just kind of left up to trust.

I don't think it was an Automattic project at first, so I doubt it's built on WordPress. But it's probably built on PHP and MySQL, with a ton of caching in front of it.

It wouldn't surprise me if it leverages WordPress.com's CDN.