I was an early adopter. In the beginning, there was actually a handful of us reviewing avatars and assigning ratings. Now it's just kind of left up to trust.

I don't think it was an Automattic project at first, so I doubt it's built on WordPress. But it's probably built on PHP and MySQL, with a ton of caching in front of it.

It wouldn't surprise me if it leverages WordPress.com's CDN.

What do you find confusing? It's owned by Wordpress guys, and people integrate it into their website. The backend stores a picture you set on the site, or one on other people's sites integrated with it. It then also keeps track of all your different accounts.