News for hackers. Stuff that matters.
Re:Web scraping is legal, US appeals court reaffirms by mindslight on Monday April 18, 2022 @06:48PM (#31077646)
> As a B2B site, all our pricing is custom as negotiated with each customer ... the pricing is dynamic

So your company is deliberately trying to frustrate the market, and doesn't like the result of third parties attempting to help market efficiency? It seems like this is the exact kind of scraping that we generally want more of! I'm sorry about your personal technical predicament, but it doesn't sound like your perspective is really coming from the moral high ground here.

Re:Web scraping is legal, US appeals court reaffirms by CWuestefeld on Tuesday April 19, 2022 @10:20AM (#31083504)
So your company is deliberately trying to frustrate the market, and doesn't like the result of third parties attempting to help market efficiency?

No. First, we as a middleman resseller MUST provide custom prices, at least to a certain degree. Consider that it's typical for manufacturers to offer different prices to, e.g., schools. This is reflected by offering to us (the middleman) a lower cost, which we pass on to applicable customers. Further, the costs and prices vary similarly from one country to another. Less obviously, many manufacturers (e.g., Microsoft, Adobe, HP) offer licensing programs that entitle those enrolled to purchase their products at a lower cost. So if nothing else, the business terms of the manufacturers whose products we sell necessitates a certain degree of custom pricing.

Second, it seems strange to characterize as "frustrating the market" what we're doing when we cooperate with customers who want to structure their expenses in different ways - say, getting a better deal on expensive products that can be classified as "capital expenses" while allowing us to recover some of that revenue by charging them somewhat more for the products that they'd classify as operational expenses.

Re:Web scraping is legal, US appeals court reaffirms by mindslight on Wednesday April 20, 2022 @10:57AM (#31097890)
You're just describing a cooperative effort to obfuscate pricing and frustrate a market. So sure, your company could be blameless and the manufacturers are solely responsible for undermining price signals. I've still described the overall dynamic that your company is participating in. It's effectively based around closed world assumptions of information control, and so it's not surprising that it conflicts with open world ethos like web scraping.

> it seems strange to characterize as "frustrating the market" what we're doing when we cooperate with customers who want to structure their expenses in different ways

I'm characterizing the overall dynamic of keeping market price discovery from working as effectively. How you may be helping customers in other ways is irrelevant.

Re:Web scraping is legal, US appeals court reaffirms by YPCrumble on Tuesday April 19, 2022 @04:12PM (#31088363)
Holy moly doesn’t that sound more like tax evasion or fraudulent accounting than financial planning?

They’re trying to pay less tax by convincing your company to put a different price on products they buy based on their tax strategy.

It sounds illegal.

Re:Web scraping is legal, US appeals court reaffirms by mindslight on Wednesday April 20, 2022 @11:00AM (#31097933)
The majority of tax and other civil laws are basically full of things that are illegal/problematic if you do them individually, but if you can find someone else to cooperate with then it becomes fine.
Re:Web scraping is legal, US appeals court reaffirms by YPCrumble on Monday April 18, 2022 @07:33PM (#31078078)
This makes a lot of sense to me.

What do you think about their other assertion that the search page is getting a gigantic number of hits that a/ cannot be cached and b/ cannot be rate limited because they're using a botnet?

Re:Web scraping is legal, US appeals court reaffirms by mindslight on Monday April 18, 2022 @07:45PM (#31078176)
I'm guessing the bots are hitting the search page because it contains the most amount of information per hit, and that the caching problems are exactly due to these dynamically generated prices or other such nonsense. After all, the fundamental goal of scraping is to straightforwardly enumerate the entire dataset.

The scale of the botnet sounds like an awfully determined and entrenched adversary, likely arising because this company has been frustrating the market for quite some time. A good faith API wouldn't make the bots change behavior tomorrow, but they certainly would if there were breaking page format changes containing a comment linking to the API.