Regardless of pullutants, we use way too much water for this to be practical. Average seawater salinity is 35 g/L, and the WHO recommends a salt intake of at most 5 g/person/day, so from one liter of water you get enough salt for seven people for a day. Approximating the Earth's population at 7 billion people, that means we could consume the salt from about 0.365 km^3 of water per year.

Just the US alone uses about ~110 km^3 of water for irrigation per year.

The water used in irrigation eventually (and pretty quickly) makes its way into the ocean, so is it actually a big deal?

The water used to irrigate crops has dissolved minerals in it. As the water is taken up by plants (or evaporated), it leaves those minerals behind. This then raises the salinity of the soil. Eventually, the soil becomes too salty/mineral encrusted to support crops. This happened in the Middle East. What is now Iraq used to be the main grain growing region of the Middle East several thousand years ago. Over time, the salt in the soil rose to the level that wheat could not grow, but barley could. Eventually, the soil became too salty for barley. Today, some of the Iraqi desert shines in sunlight because of the salt crystals. It has never recovered and will never recover. California is progressing faster.

Selenium is one of the minerals that ends up in the runoff of Californian agriculture. In some coastal areas, the selenium content is so high that bird eggs laid by wildlife never harden. Or the birth defects are so high that the hatchlings don't live.

Think about how much water a person uses per day compared to how much salt. The former is measured in liters, the latter in grams. Desalination produces far more brine than humans could ever need.