Ask HN: Dealing with a competitor who is scraping my content and ranking higher

Re:Ask HN: Dealing with a competitor who is scraping my cont... by laurieg on Monday August 20, 2018 @11:37PM (#17806761)

If you are able to detect them, serve them garbage data instead of blocking them. Mix in some false entries etc. If you completely block them they will work hard to work around the block, but if you give them some bad data randomly it will take them longer to notice.

Also, you can use the fake data as evidence they scraped you.

[ Parent | on HN ]

Re:Ask HN: Dealing with a competitor who is scraping my cont... by coding123 on Tuesday August 21, 2018 @12:14AM (#17806943)

And also to try to obtain a pattern of what request headers are common... Try to create a special hash from the request headers. Then find a way to infiltrate the data with it so you can identify them better

[ Parent | on HN ]

Re:Ask HN: Dealing with a competitor who is scraping my cont... by rakjosh on Tuesday August 21, 2018 @08:11PM (#17815020)

That is what I did initially. My API was sending garbage data Saying this website is shutting down, please visit [mysite] for more info :D. But later on, they figured, I was sending garbage data and people were visiting my site because of that. I even put in a tracked link to see how many visitors I'd get from there site and I was getting around 500 people every day.

It was like a cat and mice game for a few weeks.

[ Parent | on HN ]

Re:Ask HN: Dealing with a competitor who is scraping my cont... by acct1771 on Tuesday August 21, 2018 @02:15AM (#17807426)

It's not nearly as valuable as cryptographically provable evidence of the pages having identical content as yours, with yours being of an early publication date.

[ Parent | on HN ]