High-performance .NET by example: Filtering bot traffic

Re:High-performance .NET by example: Filtering bot traffic by senorjazz on Monday April 24, 2017 @09:57AM (#14184316)

Rather than block on UA, just add some honeypots. An invisible link. Any bot that pulls that page gets blocked as scrapers tend to pull all links from the page and follow.

Use the robots.txt to ban the pulling of specific pages. Bots 99% of the time ignore robots, so if they pull it: block

Check how quickly pages are pulled. If passes a threshold: block

[ Parent | on HN ]

Re:High-performance .NET by example: Filtering bot traffic by alexandrnikitin on Monday April 24, 2017 @10:04AM (#14184370)

Yes, using honeypots is one of the ways to identify bots. But that wasn't the focus of the post. I'll add some clarification.

[ Parent | on HN ]

Re:High-performance .NET by example: Filtering bot traffic by marklit on Monday April 24, 2017 @11:00AM (#14184799)

I've seen bot traffic claiming to be recent versions of Firefox from residential IPs in the Ukraine pulling robots.txt. Sometimes this is one of the few clues to go on.

[ Parent | on HN ]

Re:High-performance .NET by example: Filtering bot traffic by pc86 on Monday April 24, 2017 @03:33PM (#14187336)

I'm pretty sure the point of the article was performance testing of C#, not best practices for banning bots...

[ Parent | on HN ]