Client: I'm getting 504 Gateway Timeout errors on my site. Can you help?
Me: After how many seconds?
Client: About 30.
Me: Reduce the timeout to 20 seconds in your PHP-FPM pool configuration.
Client: But then I'll get even more timeouts!
Me: No you won't. While you're at it, reduce it even further to 10 seconds.
Client: What? That doesn't make any sense!
Me: Let's try 5 seconds then.
... Silence ...
Client: Hey, no more timeouts! But the widget on the sidebar is broken.
Me: That widget was trying to access an API that is currently down. Due to the long timeout and poor caching, it also took down your whole website. I heard you paid $cheapCompetitor to make the widget. If you'd like me to make you a better version by Saturday, that would be $XXX.
That's one of the reasons why Facebook has a mechanism that switches their FIFO queues to LIFO, once the server has passed a certain load/latency threshold.
That is, errors in internetworked systems can happen at a level high enough that machines can't decide if it is sane to retry or not, they should simply bubble the error all the way back to the user and let her press F5 if she wants to.
Modern UIs might hide this, but users well know the web is flaky and understand it in their bones. So when they press reload they are sending important information.
How you often see this is that your request limiter keeps booting new requests (FIFO/drop tail), and then when you check your currently processing requests, they seem to be all in some busy loop, and hopefully you have some means of checking why. You check, and you find that they're all querying a service that accepts the request but doesn't respond ever (e.g. server down behind a proxy), but for some reason the timeout on the client side is set to 30 seconds so stuff keeps hanging until it's killed.
Anyway, this is to say that in any sufficiently complex networked system scenario, it helps to review all these timeouts holistically and in the context of your application.