You can't expect the benefits of public web without bearing the costs. Just put your stuff under a auth wall (can even be free) and no one will crawl it.