Re:How F5Bot Slurps All of Reddit
by wvenable on Monday July 30, 2018 @05:23PM
(#17648171)
Except that the data stream is, I should hope, compressed so that the data is actually packed into a format optimized for size.
[ Parent
| on HN ]
Not necessarily, and even so not for free.
(Client must specify compression support)
Most clients support it implicitly; you probably have to go out of your way to get an uncompressed stream. Now compressing a verbose text string is not optimal but given the past attempts I'd hesitate against using a pre-packed format. Historically that has not worked out well. Compressing the text format is ultimately the worse-is-better solution.
They are using libcurl, for which you need to request compression explicitly: