News for hackers. Stuff that matters.
Re:Wikipedia's robots.txt by ungzd on Tuesday July 16, 2019 @04:08PM (#20453429)
> User-agent: TeleportPro

I remember using it to download whole (small) websites on dialup and then to read offline.

Re:Wikipedia's robots.txt by striking on Tuesday July 16, 2019 @07:36PM (#20454941)
In Wikipedia's case, they'd prefer you download a dump or use an app specifically designed to read Wikipedia offline. Rendering all that PHP is (or at least at one point was) expensive.
Re:Wikipedia's robots.txt by shakyshakyshaky on Wednesday July 17, 2019 @11:54AM (#20460814)
reference for above comment:

https://meta.wikimedia.org/wiki/Data_dump_torrents

I, somewhat fancifully, keep two flashdrives with a wikipedia data dump on them "just in case"

Re:Wikipedia's robots.txt by snooty on Tuesday July 16, 2019 @10:00PM (#20455759)
I didn't know they still sell TeleportPro!