Show HN: Station307 – Stream files between cURL, Wget and/or browser

Re:Show HN: Station307 – Stream files between cURL, Wget and... by hakvroot on Thursday March 24, 2016 @05:59PM (#11356829)

That's exactly what's going on. In a bit more detail: when a POST arrives and no receiver is available yet the server lets the sender know that it won't handle any more data (TCP Zero Window). When the receiver connects to the server the server asks the sender to continue writing and the data is relayed (proxied) from sender to receiver. If multiple receivers connect they're put into a queue and the server waits for the sender to POST the data again once the ongoing transfer is finished.

An interesting difference is that in the browser the rePOSTing is done through scripting whereas when using cURL or Wget (or httpie, or ...) the server will issue a 307 Temporary Redirect allowing cURL/Wget to send the file again without manual intervention. Hence the name! (although that doesn't really account for the "Station" part yet)

[ Parent | on HN ]

Re:Show HN: Station307 – Stream files between cURL, Wget and... by darkf on Friday March 25, 2016 @07:59PM (#11363679)

Hi! Thanks a lot for sharing details about it, it's an ingenious app. :)

Could you please elaborate a bit on how you work with the TCP window size -- how might I set it to 0 and back? Do you simply use SO_RCVBUF or is there more to it?

If you happen to go through with making more variants of this, it would be amazing if you could make them open source as well (even just the backend). It's a clever idea that solves a very real problem!

[ Parent | on HN ]

Re:Show HN: Station307 – Stream files between cURL, Wget and... by hakvroot on Sunday March 27, 2016 @11:56AM (#11370369)

Thank you! And I'd be happy to elaborate a bit more, although you already explained most of it yourself ;). Currently the server just stops reading from the socket and the rest is being handled by the networking stack (i.e. buffer fills up, window size is to 0). At this moment the buffer size is also fixed (it sits somewhere between having good throughput and not consuming too much memory for parked connections), but an improvement would be to change it dynamically to improve behavior on both fronts.

For any other versions or source releases I'll post an update!

[ Parent | on HN ]