It's possible that we could; my concern is our forwarding takes over IP addressing, and can forward cross-region, and that theoretically direct server return could break RPF. DSR is, of course, much easier than the flow-based response forwarding we're doing now.
I was going through Cloudflare's Magic Transit docs and they seem to simply use Direct Server Return (Facebook does so too for their L4 load balancer [0] instead of dealing with XDP in the return path).
Any reason fly.io doesn't but relays the packets again out via the edge?
[0] https://engineering.fb.com/open-source/open-sourcing-katran-...