At least when I went there it wasn't very crowded. There was a group of people at the summit (with vending machine present, of course), but otherwise hardly any people. However it was a slightly foggy day, so you couldn't see much from the summit. It is possible that on a clear day there are crowds trying to see Mount Fuji or something.