You can tune the diffuser to have an almost-instant onset. I can't remember what I did last time, but at a guess, having the largest diffuser stage increase by a factor of N (number of channels) instead of 2 might do it.

But also, if you're playing acoustic drums in a big space like a concert hall (instead of a long one like a staircase), the first echoes coming back from the walls are actually a bit delayed (1 foot ~= 1ms, at the speed of sound). So if your nearest wall is 10ft away, the first wall-echoes will come 10-20ms after the initial direct sound.