We could do this if it is important. There are mines in Wisconsin the export sand to the middle east because that is known to work well for fracking and they don't want to risk a local sand not working well. (AFAIK they have never tested local sand properties, but it is possible they have and it doesn't work). In this case the value of the "perfect" is well worth the high shipping costs.
It's all a balance. Imagine a scenario where you can ship in specialized materials to build a bridge with an expected lifespan of 100 years and it'll cost 50M - or you could use local concrete that has an expected lifespan of 15 years and materials would cost 5M. This is a vast simplification of the math but, assuming those expected costs it'd be cheaper to build using local materials and just schedule replacement every 15 years. And, of course, there'll be egg on your face if you build the 50M bridge and then suffer a massive tsunami in two years that destroys the foundations anyways.
To paraphrase a Grady quote: "Engineering isn't a study of building the best thing - it's optimizing the quality we can get for the cost outlay."
It's not really hard to test for this property, but the cost efficiency is notable when you find a massive amount of it in one place. It still may be washed to remove further silt/clay, but they absolutely know the product works and they generally know the geology in other places don't tend to produce the same material.
There's a fair amount of materials size thats mostly "we found this geologic material and this is some magic shit" rather than some wholly manufactured human endevour.
I wanted to mention that Concrete is far more complex and regional than folks might imagine. The quality of gravel and sand, local impurities - these all contribute massively. It's probably best to think of it like a wine's terroir - except, unlike a bottle of wine, it's prohibitively expensive to ship both the components and the finalized mixture to different areas. If a region's limestone has a massive clay impurity then it may simply be unsuitable for large structures or require extensive filtering to the point of being uneconomical.
It's important to be aware of just how much the local geological mix can impact the viability of building with concrete because while theoretically we could use perfect concrete for every project - at that point most projects would simply be too expensive to consider undertaking. There is a very large field of engineering around establishing the realism required in settling for what you've got for the price you can afford in. It can absolutely mean that the materials required to build a high rise in Philly might be priced starkly differently from the same structure planned in Milan even with adjustments for the labor impact on pricing.