Major factor in DRAM latency is in moving these charges around the chip (between the array and sense amplifiers). The wires in the array have some finite conductivity and current handling capability thus this cannot happen instantenuosly.

On the other hand sequential access bandwith is limited almost entirely by the external interface and has more to with economical factors (pin count is major part of IC price) than any physical limitations.