News for hackers. Stuff that matters.
Re:How F5Bot Slurps All of Reddit by yitosda on Monday July 30, 2018 @05:49PM (#17648414)
Not necessarily, and even so not for free.

(Client must specify compression support)

Re:How F5Bot Slurps All of Reddit by wvenable on Monday July 30, 2018 @06:00PM (#17648520)
Most clients support it implicitly; you probably have to go out of your way to get an uncompressed stream. Now compressing a verbose text string is not optimal but given the past attempts I'd hesitate against using a pre-packed format. Historically that has not worked out well. Compressing the text format is ultimately the worse-is-better solution.
Re:How F5Bot Slurps All of Reddit by jwilk on Monday July 30, 2018 @06:14PM (#17648636)
They are using libcurl, for which you need to request compression explicitly:

https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html