I just want to add that is very important to look at get/post and ajax calls within the site. When properly understood, they can be worked on your favor, taking away a lot of complexity of your scarpers.