Sorry to toot our own horn, but RethinkDB is a YC startup which is about as hardcore CS as you can get. Clustrix too (though you wouldn't normally hear about them since they're not too focused on the open source community). There are a couple of others but I can't think of them now off the top of my head.

Core CS/systems startups are definitely a minority in YC, but there are more than a few!