News for hackers. Stuff that matters.
Re:9th Circuit holds that scraping a public website does not... by dx87 on Monday September 09, 2019 @04:14PM (#20921961)
Would that ruling mean that sites could no longer refuse to show content based on how they're accessed? For example, sites that won't load if the browser is in headless mode, or sites that depend on javascript as a way of blocking wget/curl.
Re:9th Circuit holds that scraping a public website does not... by freehunter on Monday September 09, 2019 @04:25PM (#20922068)
I have a scraper for a site that used to offer an API for their publicly available site but removed the API with no warning. The info is still available to the general public, but only through their website. I created a scraper for the public page, but shortly after they switched to loading some public information through Javascript so my HTML scraper couldn't see it anymore. I ended up having to write an application around Selenium to load the Javascript and import this public information. I'm just waiting for them to start randomizing the CSS classes to make scraping even harder. The content is static, even as data changes on the server it does not refresh on the page unless you reload the page.

There is no reason why your page should refuse to load plain text without Javascript enabled.

Re:9th Circuit holds that scraping a public website does not... by penagwin on Monday September 09, 2019 @05:12PM (#20922534)
> There is no reason why your page should refuse to load plain text without Javascript enabled.

On a technical level sure. SPA's should pre-render data before sending it to the client.

The problem is that's a ton of extra work when the client will have to fetch data anyway - so it's difficult to justify the time to management.

EDIT: If their page fetches the data with JS you might actually have an easier time figuring what their API looks like instead of scraping the rendered page. You might find there's more data available than is rendered too.

Re:9th Circuit holds that scraping a public website does not... by leesalminen on Monday September 09, 2019 @07:36PM (#20923696)
I once showed someone how to use dev tools to see an API request a web app makes and how to recreate it with your own code. He’s since done all sorts of “life hack” automations. Love teaching people this stuff!
Re:9th Circuit holds that scraping a public website does not... by Pfhreak on Monday September 09, 2019 @05:02PM (#20922453)
> There is no reason why your page should refuse to load plain text without Javascript enabled.

Sure there is. You prefer writing javascript and you want to serve your site through a CDN.

You might not think that's a good reason, but that's certainly a reason.

Re:9th Circuit holds that scraping a public website does not... by geggam on Monday September 09, 2019 @06:06PM (#20922992)
Until the ADA comes along and demands you create an accessible to the blind site.

I've often wondered when the laws would start to be applied and I think its coming

Re:9th Circuit holds that scraping a public website does not... by tuxracer on Monday September 09, 2019 @09:42PM (#20924403)
> It is a common misconception that people with disabilities don't have or 'do' JavaScript, and thus, that it's acceptable to have inaccessible scripted interfaces, so long as it is accessible with JavaScript disabled. A 2012 survey by WebAIM of screen reader users found that 98.6% of respondents had JavaScript enabled. [0]

and that was 7 years ago. There may be certain complicated interactions that are a challenge for screen readers but simply because a page relies on JavaScript for rendering doesn't automatically mean it is inaccessible to screen readers.

[0] https://webaim.org/techniques/javascript/#reliance

Re:9th Circuit holds that scraping a public website does not... by CalRobert on Monday September 09, 2019 @06:29PM (#20923172)
I have a website that's a full page map. I care about accessibility - is there any way I can make this meaningfully accessible to the blind?
Re:9th Circuit holds that scraping a public website does not... by EpiphanyMachine on Monday September 09, 2019 @09:20PM (#20924273)
Look at WCAG 2 (web content accessiblity guidelines) they specify tags and elements common screen readers will understand to help make you site accessible.

This is a really good resource: https://accessibility.18f.gov/

A lot of frameworks now have some accessiblity built in if you add the correct attributes.

Re:9th Circuit holds that scraping a public website does not... by gnud on Monday September 09, 2019 @06:59PM (#20923429)
I suspect you're being glib, but you could look at https://wiki.openstreetmap.org/wiki/OSM_for_the_blind
Re:9th Circuit holds that scraping a public website does not... by lucasmullens on Monday September 09, 2019 @07:50PM (#20923785)
I'm confused. Why would a blind person be any less likely to use JavaScript?
Re:9th Circuit holds that scraping a public website does not... by geggam on Monday September 09, 2019 @07:53PM (#20923802)
To test for a blind persons ability to render your website a good method is a CLI browser. Neither the blind persons device or a CLI browser will render javascript
Re:9th Circuit holds that scraping a public website does not... by zaroth on Monday September 09, 2019 @08:21PM (#20923978)
I’m no expert in this space, but if free, common, and easily accessible tools render your site readable and usable, I’m not sure how refusing to use those tools would be a claim under ADA.

There are plenty of real ways that sites are unusable by screen readers, using Javascript to download dynamic content shouldn’t be one of them.

Re:9th Circuit holds that scraping a public website does not... by TylerE on Monday September 09, 2019 @11:55PM (#20924913)
Why on earth would you think that? The screen readers tie into modern browsers like Safari or Chrome.
Re:9th Circuit holds that scraping a public website does not... by Franciscouzo on Monday September 09, 2019 @05:29PM (#20922653)
What does a CDN has to do with it?
Re:9th Circuit holds that scraping a public website does not... by Spivak on Monday September 09, 2019 @05:41PM (#20922759)
Because if your application bundle is a fixed asset — like a JS SPA that fetches it’s data from an API then you can distribute your entire application via an inexpensive CDN.

As soon as your application bundle is rendered on your servers dynamically then only part of your site can be delivered via CDN.

Basically going all-JS gives you an app model where your sever side code doesn’t even know or care about HTML or the web or whatever. It just pushes JSON or whatever around and it largely client independent.

Great model when you need to support iOS, Android, web, desktop.

Re:9th Circuit holds that scraping a public website does not... by penagwin on Monday September 09, 2019 @05:44PM (#20922794)
Using a CDN like their talking about likely means your html is static and only served from the CDN.

For many SPA's the only actual html is a header, container div, and a call to the app's js. Ignoring the header there might only be say half a dozen lines total.

Re:9th Circuit holds that scraping a public website does not... by mirimir on Monday September 09, 2019 @05:48PM (#20922841)
Chrome headless does that really well.

> google-chrome --headless --run-all-compositor-stages-before-draw --virtual-time-budget=25000 --print-to-pdf='foo.pdf' URL

Edit: Plus

> pdftotext -raw foo.pdf

Re:9th Circuit holds that scraping a public website does not... by paulcole on Monday September 09, 2019 @08:39PM (#20924077)
> There is no reason why your page should refuse to load plain text without Javascript enabled.

Of course there is! It’s my site and I can do what I want with it.

Re:9th Circuit holds that scraping a public website does not... by slg on Monday September 09, 2019 @06:08PM (#20923018)
Captchas would be another technology that might be fall under that "technical barriers" terminology. I don't do much scraping, but I think most of us would enjoy never having to "identify the traffic lights in these photos".
Re:9th Circuit holds that scraping a public website does not... by TeMPOraL on Tuesday September 10, 2019 @03:37AM (#20925899)
I would very much hope so. These sites are a perversion of the idea of the Web, and only get away with it because fundamentals of the Web were created with friendly, cooperative users in mind - not with commercial users of today - so there's no enforcement of good behaviour.
Re:9th Circuit holds that scraping a public website does not... by otterley on Monday September 09, 2019 @04:30PM (#20922133)
We don’t know yet. It would depend on the specific legal question that is decided in the case. Courts usually try hard to constrain the law to as few questions as they need to in order to resolve a dispute.