The second stage debris that you are talking about showed up when the stage stopped accelerating. The debris and stage were moving at the same speed, so it's normal that they drifted apart relatively slowly.