Yes, this is huge, and likely one of the least appreciated aspects. Scheduling and queueing delay (not in the switches and routers) can contribute significantly to that 100ms.