The way I've had it described is that when two objects of the same shape pitch for optimal glide (i.e. highest L/D) then the heavier one will reach the ground sooner (go faster), but both will take the exact same path and land the same distance away. In other words, same L/D.

This is not the explanation you are looking for, but "aha the heavier object takes the same path but drops faster" was what made me okay with L/D not depending on weight.