An example is that I do use Redis for some queueing operations. But there's a bottleneck in part of the process (not with Redis, but the nature of being unable to POP multiple items off of a list at once using the blocking pop requests, requiring multiple requests over a socket, or two separate data stores; one to notify, another to hold the data). So I'm in the process of writing my own queue for the crawler system.
I'm guessing that your question comes from a curiosity or desire to build something similar, or something similarly performant. I don't know if this is a useful perspective or not, but it's mine: that every time I don't know what's happening to a byte of data and how it's processed, there's the possibility of some unknown amount of inefficiency there, and that inefficiency could be small or large. But the not knowing is the problem. So I've tried to eliminate as many of those as possible. I don't always get it right, and I'm sure there are many areas for improvement. But if I find a performance bottleneck, I go in and edit my own code, rather than having to learn someone else's codebase that is written to handle edge- and use-cases I'll never need. I then either have to modify their code - and have to maintain future updates as well, etc - or I write my own, and keep refining it.
I'm certainly in no position to give any sort of programming advice to anyone, but that's just the philosophy I've had from the beginning while I worked on this. And I think it's an important one that's helped me get the performance I've achieved with it. It's a conversation I have often with other friends who are programmers and who make more liberal use of tech stacks. But those are often problems of a very different nature than the ones I'm interested in solving.
And if you do hope to build something similar, or something different but similarly performant, be sure to test everything. You don't need amazing tests, just consistent tests with consistent units of measure. Learn where the compiler is optimizing things out and how to make sure that doesn't happen, etc (I will often insert an integer that gets incremented by a random number generated by a fast xorshift prng at the critical place of measure, in order to increase the odds the compiler won't optimize it out), to ensure a level of quality in your tests. And when you're working at the byte level, keep refining the operations that run the most frequently and test all possibly ways you could imagine doing it.
As for the rate of requests, I just couldn't figure out how to max out the gigabit connection on Linux with epoll. I ended up moving to FreeBSD and kqueue, and when I did, that's when I was finally able to comfortably max out the connection. I could have just been doing something wrong with epoll, but I wasn't figuring it out, and neither did a friend who looked at it. My guess is that some of the limitations of the api that are resolved in kqueue, just weren't cutting it for that much data coming into epoll.
I briefly considered writing my own tcp stack to run on raw sockets, but I'm quite happy with FreeBSD's network stack, which seems to be more than sufficient.
As for the size of the project, toward the end, it kept shrinking as I better eliminated redundant logic. There might be more ways to improve it, but I'm pretty happy with where it is.