News for hackers. Stuff that matters.
Re:Robots.txt for the NYT has a specific exclusion for an 19... by DyslexicAtheist on Monday March 29, 2021 @06:27AM (#26619694)
the US Department of State (DoS) during 2012/2013 in its robots.txt[1] excluded around 9577 documents which leaked into archive.org (already pre Snowden). The robots.txt file now is OK but not sure if content is still on archive.

[1] https://pastebin.com/raw/RE2tpyR3

  8<-----------8<-----------8<-----------8<-----------8<-----------

  #!/bin/bash
  snapshots="20120713050942 20121013154343 20121010165822 20120921054221 20130413152313 20130113162428"
  # orig source http://state.gov/robots.txt but also on pastebin in case they delete it:
  wget --output-document=robots.txt http://pastebin.com/raw.php?i=RE2tpyR3
  for x in `echo $snapshots`
  do
    for i in `cat ./robots.txt|cut -d ' ' -f2 | tr -d '\15\32'`
    do
      if [ -e `basename $i` ]; then
        echo "$i already fetched"
      else
        wget https://web.archive.org/web/$x/http://www.state.gov/documents/$i;
      fi
    done
  done

There are no comments on this comment yet.