Games are applications designed to solve a particular problem domain, no different than any other type of software. It just happens that the problem they solve is "having fun" or, more cynically, "monetizing fun."

Most "game developers" don't even ship anything (currently including myself) so what I would be concerned about is not having a game, which itself puts you ahead of a lot of people, but what the quality of the game is, and what it demonstrates about your abilities as a programmer. Games can be incredibly complex and difficult to build and can demonstrate a myriad of useful skills. As someone with a CS degree in, basically, "C# CRUD apps that talk to Access" who dabbles in game development on the side, I definitely respect it. Even using a framework like Unity doesn't make it easy, just easier.

Unfortunately (although somewhat deservedly) Unity development and indie game development so there might be a bit of friction involved if that's all you have in your portfolio.

But no, I wouldn't remove them, certainly not if they're any good.