The surface of the moon is dramatically more hostile than the surface of Mars by every metric, and it's almost as hard to get to the moon as it is to get to Mars. Building a long term permanent colony on the moon would be far more difficult than doing so on Mars.

As long as you can manage the travel, Mars is by far the more desirable target for colonization, both short term and long term. Elon Musk believes that his ITS can cut travel time down to 2-3 months, which seems reasonable, even if your craft's shielding isn't the greatest.

Can you give some examples for how the moon is more hostile? I've always thought that for both you need a pressure suit but that the temps and radiation numbers were similar? Sure it's 21 days of hot then 21 days of cold instead of 24.5 hours but that seems much easier to deal with than the communications delay and travel times?

The long days + nights pose problems beyond just temperatures. It means having to store a lot of energy to deal with the long nights, meaning either big batteries (heavy and hard to get out of Earth's gravity well) or nuclear reactors (riddled with political issues). The 24.5 hour days on Mars can easily be handled with small batteries, or if need be power generated by fuel extracted on Mars.

Gravity on the moon is also a big issue for long term habitation. It's only 16.7% of Earth's gravity, which is a far cry from Mars' 40% and is much more likely to cause physiological issues with the human body. 40% gravity may be enough for negative effects to be mostly offset by exercise, but 16.7% is much more doubtful. Crews would likely need to be regularly cycled, making it impossible for anybody to live on the moon permanently.

The surface of the moon is exposed to much higher levels of radiation by two counts: first, it's closer to the sun, and two, it has no atmosphere. Mars' surface radiation is a good deal lower thanks to extra distance and its atmosphere, as thin as it is, cuts down on that number significantly. Furthermore, with 24.5 hour days there are frequent breaks from exposure to solar radiation whereas moon colonists would be faced with 21 days of high exposure followed by 21 days of low exposure.

There's also the matter of resources. Raw material is both far more plentiful and more accessible on Mars; there's an atmosphere to pull gases from for oxygen and fuel and entire lakes of frozen water on Mars, whereas moon settlers would need to use expensive, complicated, and failure-prone machinery to process regolith. There are craters with some frozen water on the moon, but relying on those greatly limits the number of prospective colonization sites and will eventually be exhausted if population counts rise from outpost numbers to something more closely resembling a permanent colony.

There are other factors as well, but these four are some of the largest.

In addition to kitsunesoba's excellent answer, consider the potential hazards of lunar dust: https://www.sciencedaily.com/releases/2008/09/080924191552.h... .

That isn't to say that Martian dust won't be an issue, of course...but perhaps one that is slightly less problematic.