For every legitimate use of binaries in source control there are dozens of idiots checking in build artifacts.
Build artifacts aren't necessary to bring over to git in a conversion from hg/svn. I archive installers and other artifacts in svn, but if I moved to git I'd just archive those somewhere else. The binaries that are actually part of the build (a few binary dependencies, lots of graphics, test in/out data etc) must be next to the source however because those are needed to reproduce a build.
(Tip for Unity users: The very first thing to do when you start a project is to go into the project settings and change it to Text Assets. That way the scene files and such will be text instead of binary. It's still a fairly opaque YAML format, but at least they will be more diffable.)