News for hackers. Stuff that matters.
Re:Upton: A Web Scraping Framework by praveenhm on Monday July 22, 2013 @05:26PM (#6086563)
Is web scraping legal?
Re:Upton: A Web Scraping Framework by martin-adams on Monday July 22, 2013 @05:31PM (#6086594)
It can be, I've just launched https://myshopdata.com where online retailers can scrape their own content for synchronising with third party marketplaces.
Re:Upton: A Web Scraping Framework by darrhiggs on Monday July 22, 2013 @05:42PM (#6086653)
Your app looks really good.

What tech did you use if you don't mind answering?

Re:Upton: A Web Scraping Framework by martin-adams on Monday July 22, 2013 @05:55PM (#6086720)
Thanks!

My Shop Data is all PHP & MySQL, with Slim, Twig & Bootstrap. The web scraping aspect is another product of mine (forgive the clunky homepage, I'm going to turn this into an API platform) - https://grabnotify.com

GrabNotify is Node.js, Mongo, PHP, Bootstrap and PhantomJS. The undocumented API allows you to create a web crawler but define a JavaScript algorithm to extract the data off the page. Some retailers have dropdowns which update stock, images, etc, so this crawler can simulate mouse events, etc. My Shop Data will supply a custom crawler algorithm for each e-commerce web site through the API.

And finally, I've written a HTML to Markdown translator to extract page descriptions but keep some formatting while being transferable to other systems that don't support HTML.

The whole legality issue of web scraping is an interesting one. I'm planning to position GrabNotify as a web crawler, page monitor and HTML -> data tool, but only if you own or have permission to scrape the original content but need a simple way to grab and monitor the HTML into data. I'm not really interested in building a business of scraping other people's content without their permission.

Re:Upton: A Web Scraping Framework by adamnemecek on Monday July 22, 2013 @05:32PM (#6086597)
If anything, it might be against TOS but ASAIK, breaking TOS is not illegal.
Re:Upton: A Web Scraping Framework by lsiebert on Monday July 22, 2013 @05:56PM (#6086726)
Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.
Re:Upton: A Web Scraping Framework by adamnemecek on Monday July 22, 2013 @06:17PM (#6086830)
Re:Upton: A Web Scraping Framework by tlrobinson on Monday July 22, 2013 @06:05PM (#6086769)
Is there's no signup/login process is the TOS even enforceable?