For example, I live in northern New England, which is heavy on trees and light on sand.

That's why our typical "vernacular architecture" is log cabins, not mud brick structures. Different advantages and disadvantages. Different construction process. Different materials. But still more sustainable than concrete.