We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site.
Some sites are just so damn good and detecting us and I just don't get it.
The countermeasure would be to have a bunch of humans use the websites in any way they want, totally undirected, then use the totality of that browsing to facilitate your scraping probabilistically. It would be less efficient, but very difficult to catch.
Given that your run this division there is a good chance you are personally liable.
We've had this division for many many years, and before my time we paid another company to do this. There's no legal issues.
Computer security laws are very broad. It doesn't matter if it's just a website that the public can access. If you're accessing it in a matter that they don't want AND you're aware of that, then I struggle to see how your lawyers can justify it.
> Computer hacking is broadly defined as intentionally accesses a computer without authorization or exceeds authorized access.
https://definitions.uslegal.com/c/computer-hacking/
Hiding your user agent because you know they don't want automated retrieval of information is "without authorisation".
Don't think connecting a computer to a private network to suck up subscriber data is comparable to scraping publicly accessible internet content.
First, the accuser needs to, at least, send a cease and desist letter to the accused asking them to stop accessing the protected computer. Second, the accused needs to ignore that request and keep accessing the protected computer.
Is it possible to build a solid CFAA case when those two things do not happen? I cannot find any examples.
https://iapp.org/news/a/can-a-cease-and-desist-notice-create...
Even if you aren't trying to disguise anything, adding some randomness helps avoid one particular bad pattern with operations on a network. I recall the pattern being called "network synchronization" but I can't get good search results for that.
I'm curious why you'd jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It's just very difficult to make an automated scraper look like a residential or business user.
Where do you run your scraping operation? Is it in AWS or some other hosting provider, because that will get you blocked quickly by a lot of sites? Do you rate limit, including adding random jitter to mimic the way a human might use a browser?
There's scraping services available that essentially use a network of browsers on residential connections with their extension installed to get around scraping detection. It's much slower, but it's much more reliable. We also had some success by signing up with a bunch of the VPN providers (PIA, NordVPN, ExpressVPN, etc) and cycling through their servers frequently. Anything to avoid creating patterns that look automated or being tied to an IP that can be blacklisted. I'd start there before I'd worry about hacky javascript detection like in this story being what's tripping you up.