Concrete and brick are a natural, inert, substances that have been present, in some similar form, in nearly all bodies of water, and most natural enviromnents, since the beginning of life. Plastics have been around for 113 years, and they don't quickly remove themselves from the water ways (they're buoyant), as rock dust/sand does.

The rate of decomposition is not the correct metric to look at, in this context.