raw system call overhead: this isn't alot, but in an i/o heavy application it shows up as a reasonable percentage of work.
general purpose network machinery: things like kernel thread context switches for interrupts and networking likely wouldn't be there in the more bare bones systems. also none of the firewall/packet filter/netgraph checks will be there. its also possible that a single application network service might revert to a pure polling model and avoid interrupt context switches entirely.
unix interface: your standalone web server interface might be very well written to not use the copying tcp read(), and use direct callbacks or thread handoffs instead of select/poll/kqueue
other system gunk: none of the usual daemons and their kernel state needs to be there, and they wont be waking up and stealing cycles from your service. you should expect a much tighter service latency
another factor at play here is that alot of the general OS paths have had substantial work on them (interrupt coalescing, thread scheduling, memory management etc). that kind of effort may not be replicated in the simple standalone version. so by throwing that away you're losing some of the good along with the bad (where the bad here is otherwise useful generality, not something really objectively bad)
as mentioned in the HT writeup, startup time can be vanishingly small if thats of importance to you.
overall there should be some modest performance gains, but really not enough to drive adoption solely on that basis. for me the real win is radically simplified management.
given that everyone is building/using management layers over OS instances in a cloud context - is it really useful to carry around that large amount of additional machinery to manage the per-OS configuration? if I can trivially build and deploy new instances and wire them into my service, do I need chef anymore? do I need to fuss with user permissions in a world where no one is really logging in?
why manage a firewall, which assumes that all sorts of random junk might be running and we need to implement a policy for safety. with tools like this I can construct a system which by construction only responds to requests for the service of interest.
do I need to maintain a shell environment for monitoring and debugability? maybe - but effective distributed deployments already use distributed monitoring and tracing tools that are arguably more effective - even for general purpose OS deployments.
Of course, you’re throwing away a lot. But for certain applications (like HFT), the potential benefits seem very attractive.
How would the performance ande memory usage of those three things compare?