There's a balance there. Personally, I like how Link to the Past handled that: some walls have cracks in them, some (but not all) cracks can be bombed open into doors, but you can always tell the difference by tapping the wall with your sword and listening to the sound.