> Mercury itself is light enough to cancel that out.

No, it does not cancel it out.

You need about 12km/s delta V to get to Mercury from Earth and then the same to get back. Remember, if you need to lift anything from Mercury, you need to actually transport fuel TO Mercury, before you can use it to lift anything out of Mercury.

And you don't just need twice more fuel for twice more delta-V. The fuel requirements grow exponentially.

Now imagine, how poor idea it is to transport mountains of fuel to Mercury just to get a tiny bit of what exactly? Raw materials? What kind of raw materials are present on Mercury that would warrant it?

It depends on your destination. It doesn't make sense to get things on Mercury and bring them to Earth, but if you need them in space, it may make sense. Don't forget that you pay about 12-13 km/s of delta V to get anything from Earth either.

You still don't seem to understand. You need 12-13km/s to get to anywhere useful from Mercury. Closer to Sun than Earth there is only Mercury and Venus and nothing else.

For example, if you wanted to build a base on Mars using resources that you obtained somewhere else, it would actually require many times more fuel to get anything from Mercury than it would be to get it from Earth surface.