I agree that a transport cannot hide all possible failures and that at some point things have to get kicked back to the application. But if the only guarantee provided by the transport is byte delivery in one direction, that makes things significantly harder for apps: every app now has to implement its own timers and pings to detect server failures. This basically duplicates timers and retries already present in the kernel. I think Homa's approach makes life easier for apps. There's a clear and simple division of responsibility. Homa detects all failures, including both lost packets and server failures. It recovers from packet loss but not server failures. Server failures are reported to the app, and the app decides how to handle them.
And, Homa's approach cuts the number of packets in half in the common case of short RPCs with fast service times. With acks it takes 4 packets for each RPC: one for the request packet, one for its ACK, one for the response packet, and one for its ACK. With Homa there are only 2 packets in the common case: one for the request and one for the response. The response implicitly acks the request, and the next request acks the previous response.
Hmmm, I see now that QUIC can delay ACKs and piggyback them on subsequent data packets, so it looks like it can get similar efficiencies to Homa.
I agree that Homa holds Tx resources for longer than would be needed in an ack-based scheme (I don't see that as particularly problematic).
In a lossless RPC the client should expect the reply in 1002 ms. When should the client resend the request if it got lost?
Given that the client would not expect to see the first byte of the reply until 1002 ms and receives no other feedback until then, they should not issue a resend of their request until 1002 ms.
That means if the network lost their first request, they will not send a resend until 1002 ms and then get the reply to their resend at 2004 ms.
In contrast, in a ack based approach, the client would expect a server ack in ~2 ms as the server gives feedback on receipt rather than on processing completion/reply start.
That means if the network lost their first request they will not send a resend until 2 ms and then get the reply to their resend at 1004 ms, nearly half the latency.
In ack based approaches, packet loss requires one additional RTT latency to recover per repeated packet loss. But Homa incurs one additional RTT + processing latency to recover per repeated packet loss.
As a secondary downside, this requires the client to hold Tx resources for RTT + processing time since the client does not know when the server has received all of the data. In ack-based approaches, you learn the server has received all of the request when it acks all of the data in the request which is on the order of the RTT and thus can release Tx resources at that time. This is similar to the reason Homa has ACK/NEED_ACK packets except applied to client side resources instead of just server side resources.
The case you are describing is more like what if the server explodes, but you still need the results/modifications related to your RPC to occur. In that case, whether the Tx got through or not is irrelevant because your RPC still did not complete and you need it to complete, so there is no advantage to the ack-based approach. But by that logic, what if it explodes before persisting changes? Or what if all your servers explode? At some point we should call that out of scope for a transport protocol.
I prefer setting that line at “bytes got from one end to the other”. If you want higher guarantees then you can layer that. As you point out, it is easy to just layer a “server dead” timeout ping which basically solves the problem in the same way Homa needs to.