I'm assuming here just based on experience, but...you don't just go to your seat and sit down, you have to put your bag in the overhead bin, which takes some "aisle time".

in back-to-front, a line forms as you are waiting for people to store their bag. one person can take up [block] 3 rows of seats while he is fiddling with putting his bag overhead.

in random seating, people all over the plane are storing bags at the same time, instead of just 1 or 2 people.