Re:On incomplete HTTP reads and the requests library in Python
by batbomb on Sunday April 22, 2018 @04:55PM
(#16898524)
Isn’t the chunk the unit, why should it be unbounded m? You’d have issues on the client side without knowing how long a chunk is going to be - It seems like the chunk should be the unit which can be guaranteed to fit in memory on the client.
Re:On incomplete HTTP reads and the requests library in Python
by cryptonector on Sunday April 22, 2018 @05:59PM
(#16898845)
Chunked transfer encoding means that the sender sends <length>\r\n<chunk-of-that-length>\r\n... but each chunk can be of different length. A zero-length chunk terminates the transfer.
Chunked transfer encoding is used when the sender doesn't yet know the length of the resource's representation.
Re:On incomplete HTTP reads and the requests library in Python
by greglindahl on Sunday April 22, 2018 @08:45PM
(#16899670)
You could consider using websockets, which are intended to be used the way you're trying to use chunking.
Re:On incomplete HTTP reads and the requests library in Python
by cryptonector on Monday April 23, 2018 @09:51AM
(#16903368)
I'm aware. But this is easier to use. For example, here's how you tail some log file: curl -H 'Range: bytes=0-' https://foo.example/bar.log.
Re:On incomplete HTTP reads and the requests library in Python
by zie on Sunday April 22, 2018 @05:15PM
(#16898628)
Is this OSS? Is it available for use anywhere?
Re:On incomplete HTTP reads and the requests library in Python
by cryptonector on Sunday April 22, 2018 @05:56PM
(#16898828)
Not yet. I've asked permission to open source it. It's basically epoll-based, evented, and it's darned simple.
This is incredibly handy, both for the usual things one might want to tail -f (log files), and as a cheap-but-very-functional pub/sub system.
One thing I've learned is to watch out for silly proxies (client or reverse) that want to read the whole thing and then re-serve it as definite-length (i.e., not chunked, with Content-Length) or which impose a timeout on the transfer.
HTTP needs a way to say that a chunked encoding is of indefinite length, though arguably the Range: header in my case ought to be all the hint the proxies (and libraries!) need.