It seems it has been used in both and other ways: https://dictionary.cambridge.org/us/example/english/fabric-o...

For what is worth, your comment and this link[1] is the first time I have seen it used for material infrastructure.

[1] I am on the fence whether the quotes in the link can straightforwardly be interpreted to be about material infrastructure as opposed to "moral/social fabric".