No, I'm talking about handling requests. In this particular case, requests (32 to 64 bytes) were flowing through several services (on the same computer). I replaced the processing chain with a single application to remove the overhead of serialization between processes. Requests were filtered early in the pipeline, which made a ~55% reduction in the work needed.
Requests were then batched into succinct data structures and processed via SIMD. Output used to be JSON, but I instead wrote a custom memory allocator and just memcpy the entire blob on to the wire.
Before: No pre-filtering, off-the-shelf databases (PSQL), queue system for I/O, named pipes and TCP/IP for local data transfer. Lots of concurrency issues, thread starvation and I/O bound work.
After: Agnessive pre-filtering, succinct data structures for cache coherence, no serialization overhead, SIMD processing. Can saturate a 32 core CPU with almost no overhead.
[1] https://www.techempower.com/benchmarks/#section=data-r13&hw=...
Let's say I build a sorting algorithm that is O(N^2) complexity and works fine for small inputs (takes <1 millisecond), but it is going to be used for large data systems. Suddenly it takes hundreds of thousands of hours to sort the data.
One of the corps I worked with went full scalability in their architecture. One-click deployments, dynamic scaling of servers, rebalancing of databases, automatic provisioning of storage. They were handling 40-50k requests pr. second with their 15-ish large server farm, which could sale down to 5 servers, or up to 50-ish before it began to wobble.
I got called in because the company had gotten a large client that needed 100k requests pr. second. They tried scaling the system to fit the need, but the whole thing got unstable and their solution was "more operations people to manage it".
I built a custom solution for the backend. Took about two months. The new system could do about 2100k requests pr. second on one server. Scalability of the new system was ~90% efficient as well, so lots of capacity for the future.
None of their developers understood computers or the science behind them. They were all educated and experienced developers, but none of that were applied to the problem. They were just assembling parts from the hardware store until something worked, and the resulting Frankenstein's Monster was put into production.