And because you don't know whether it is damaged or not (sometimes the damage may not be obvious), it's likely that the rocket will have to get a long post-flight inspection to check if it is suitable for another flight. It's something you can almost completely avoid if you land the rocket gently.
I read somewhere, that the costs of recovering SRBs of Space Shuttle from the ocean and then inspecting and fixing them were many times greater than building another pair of boosters.
Also, part of the allure of SRBs is that they are cheap to manufacture, comparatively (this is a false savings, due to increased operational complexity, but it's still very tempting), so even if a significant amount of money could be saved per SRB through reuse it wouldn't have affected the cost a launch much.
It's more like wanting the companies that make them more profitable by requiring no retooling. I don't buy the "it's about the workers" thing.
The cost of launching a Shuttle including the amortized development costs ended up being $1.5 billion per launch.
If you need more payload build a bigger rocket.
You could also add wings to the rocket and fly it home which has it's own set of trade-offs and benefits(see space shuttle).