There's a whole strand of veganism called bivalveganism which supports the usual vegan diet + eating bi-valve creatures like mussles precisely because their cultivation seems to have a lot of positive environmental benefits (or at least, it's one of the least harmful, as you mentioned) and because they seem to rank pretty low in terms of sentience / capacity-to-feel-pain.
> Crop agriculture is however the most common source for water pollution
Again I think the most compelling rebuttal is the fact that dairy/meat is the primary driver of crop agriculture today. We can feed the world on literally a fraction of the crop agriculture that we currently do. The vast majority of that crop agriculture goes towards feeding the animals we use for dairy or meat. So in terms of thinking about the whole picture, as you mention, any problems with crop agriculture are exacerbated by meat/dairy reliance.
Thus why I go for "the whole picture" approach. Locally produced food with markings for organic and sustainability is usually devoid of burned rain forest. Small producers tend to value sustainability more than large factory farms. Crops in season tend to involve less obscurity and less complex process which can hide ecological crimes. Animal farms with fewer animals tend to care more about individual animals health than larger farms that treat animals as items.
A big reason why organic crops has a rather complex picture comes from the issue that there exist no free lunch. Farmers that do not use chemical fertilizers derived from natural gas will instead use natural fertilizers. What that actually mean from a ecological perspective is that the chemical fertilizers produced from natural gas get put in the ground to produce animal feed, the animal feed get put into animals, and the rest product in form of manure get sold as a natural fertilizers which then is used to produce organic crops. Since organic farmers need to use more manure than non-organic farmers, and the output is lower, the total amount of carbon emissions per product can be argued as higher depending on how one count and attribute emissions. Still I generally prefer organic over non-organic because it does not directly put natural gas into the ground, and I find the cost in increase land use preferable over the other trade-offs.
Sustainable grazing (most grazing isn't) is a perfectly good use of marginal land, but that cannot produce anywhere near the meat we consume. Most animal production requires producing feed from other agricultural products. It is inherently less efficient to do this. Chicken is more efficient than beef, but still much worse than just eating plants directly.
For example, freshwater consumption is important in regions which lacks freshwater. Crop agriculture is however the most common source for water pollution, occurs in all regions, and is not only a major environmental problem but also harms the supply of fresh water even if its not responsible for the primary consumption. Fertilizers and pesticides being the main culprits here. Sadly there is very little food in stores that do not have a direct link to fertilizers and pesticides except for wild fish, shellfish and seaweed.
In quite a few times I have seen studies showing that the lowest carbon emission in any food group would be either shellfish, seaweed, or insects. No fertilizers, pesticides, herbicides, minimal land usage and sustainable. Vegan would exclude two of those, and seaweed is pretty rare in non-asian diets.
In general I try to look for marks of sustainability when buying food. Small producers, non-factory farm operations, local, crops that are in season, and so on. The article here focus on the issue of sex determining the eggs, but my primary priority is the area that the hen has. In EU you can have 16 hens located in a small box the size of 0.2m². That is plain cruelty and so I choose eggs under the mark that require 4m², a requirement for outside area, always access to natural lighting, and given crops grown without chemical fertilizers, pesticides or herbicides.
Is it perfect? No. Not using fertilizers, pesticides, and herbicides is still being debated and researched if they are better for the environment or worse since land usage increases from it, but since I regularly dive in the Baltic sea I am constantly reminded by the harm done already by the revolution of chemical fertilizers produced primarily by natural gas.