News for hackers. Stuff that matters.

A comment on Wikipedia's robots.txt. Parent comment

Re:Wikipedia's robots.txt by spc476 on Tuesday July 16, 2019 @06:28PM (#20454453)
You can read the rest of my MJ12Bot saga: http://boston.conman.org/2019/07/09-12 My take: they are grossly incompetent at programming.
[ Parent | on HN ]
Re:Wikipedia's robots.txt by Double_a_92 on Wednesday July 17, 2019 @02:42AM (#20457088)
Turns out that if one wants something from people, one shouldn't start the first interaction by calling them "grossly incompetent"...
[ Parent | on HN ]
Re:Wikipedia's robots.txt by syxun on Tuesday July 16, 2019 @07:46PM (#20455003)
The best robots.txt for Majestic:

    iptables -A INPUT -s 207.244.157.10/32 -j DROP
[ Parent | on HN ]
Re:Wikipedia's robots.txt by spc476 on Tuesday July 16, 2019 @10:20PM (#20455842)
If only it was that easy. Last month MJ12Bot hit my site from 136 distinct IP addresses. If we drop the last octet, it's 120 unique class-C addresses, and if we drop the last two octets, then 43 unique class-B addresses (and why not---31 distinct class-A addresses). It's a distributed bot. Very hard to block, so I think I came out ahead by them no longer spidering my site.

Edit: Added count of class-A blocks.

[ Parent | on HN ]