dmn001

I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.

By caching, I just mean storing of data locally so you don't have to request it again under a certain timeframe. I use my own caching scripts written in Python, if you use a 3rd party library then data deletion does not matter too much either if you configure it properly and backup the data - html/json data compresses really well using lzma2 in 7-zip.

There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging.

A project like Scrapy should have caching on by default, but it seems to be an afterthought. Repeatable and reproducible parsing of cached websites is necessary, e.g. if you find additional data fields that you want to parse without downloading the entire site over again.

On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.

That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.

SVG flowcharts of many gamebooks including FF and Lone Wolf: http://outspaced.fightingfantasy.net/SVG_Flowcharts/main.htm...

Should be fixed now.

I made .svg diagrams for the Fighting Fantasy books Warlock of Firetop Mountain and Deathtrap dungeon a while back on my old blog: https://daveman.wordpress.com/2010/01/08/how-to-create-svg-l...

No. Most websites don't do this.

It's extremely rare to be ip-blocked by any website just for using the Google's user agent from a non-specific range. IP's get re-used and you can switch to a new one easily, so it's really not common or good practice for this to happen.

SEEKING WORK - England,UK/Remote

Over 7 years experience with Python software development, cloud services, data mining, web crawling, databases. Want to extract or crawl data from a website such as business listings, sports data, government data, site directories, either one-off, periodically, or in real-time? Contact me via email: dmn001(at.[gmail

The first part seems like a very long-winded way to say "don't use the default user agent".

The captcha was unusually simple to solve, in most cases the best strategy is to avoid seeing it in the first place.

Right, if you're just matching a substring in a list of strings then you don't need regex for that.

Also, lxml is over 20x faster than BeautifulSoup, although it doesn't really matter with such a low frequency of requests, it's something to note if you had another project that required parsing e.g. 1,000's of html pages.

SEEKING WORK - UK/Remote

Over 7 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.

I don't think I have the other videos, IIRC he removed or made private the previous videos shortly after the later ones were uploaded and that was the impetus for downloading them in the first place.

Btw, here is an another video of Gatto in the BJC uploaded very recently that I'll think you'll enjoy: https://www.youtube.com/watch?v=gtsaTMdKVTM

I found some of Vova's videos:

https://www.youtube.com/watch?v=_zG_QpwbV3E

https://www.youtube.com/watch?v=9csBR4AcSAQ

https://www.youtube.com/watch?v=XfGbWbWSr00

https://www.youtube.com/watch?v=_08RmjLVrpE

I think I have Gatto's videos saved somewhere on my hard drive, I'll try and upload them somewhere if I can find them.

edit: found one of them, I think there was another video before this: http://juggling.tv/16257

brief timeline of events: http://www.galchenko.com/oldsite/news,_2008.htm http://www.jugglingsubculture.com/?p=112 https://groups.google.com/forum/#!searchin/rec.juggling/5up3... https://groups.google.com/forum/#!topic/rec.juggling/OcYsxJC...

It would just be much easier to use the Facebook graph api, there is an official Python module and is well documented, and would be less likely to hit rate limits or other blocks - ironically that was one of the reasons that the author used scraping instead of the api.

SEEKING WORK - UK/Remote

Over 5 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.

Location: UK

Remote: Yes.

Willing to relocate: Yes.

Technologies: Python, Perl, Matlab, R, MongoDB, MySQL, Git

Résumé/CV: available on request

Email: dmn001 gmail

Btw, I have no association with this company, other than I like to keep track of their blog and events regularly as it ties in quite closely with my line of work. Thought I'd post as it's an article that fits in the startup culture of being able to adapt quickly and dealing with situations you can't prepare for, whilst turning them around into a positive.

SEEKING WORK - UK/Remote

5+ years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.

> the entire content of The Pirate Bay > Not the P2P content that people download

I think you're confusing different definitions of 'content' here. Also, there are many more sites that have publicly available indexes of other links much greater than the pirate bay, google.com for example.

Furthermore, 8GB is relatively small for a flash drive nowadays, e.g. here is an 128GB micro sd card: http://www.amazon.com/dp/B00M562LF4

SEEKING WORK - remote or UK

5+ years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Need to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.

Yes, I heard that one is expensive. Just out of interest, what are you using it for?

Is there any documentation for that api, from the url it looks that they created it specifically for their official mobile app only?

Other browser based screen scrapers that are in the space are 80 legs, kiminolabs, Mozenda and OutWit Hub, I'm sure there are more. Last time I checked, import.io was a fairly lightweight browser wrapper.

I also write web scrapers using Perl and Python, recently have been gravitating towards Python as the code looks more readable. I don't use browser based scrapers because the sites I scrape are usually more complex so it is just easier to write my own code, and they lack functionality and control of the data, and there is the overhead of learning the terminology and how it works.

Looks like it just uses browser-side js requests to get the search pages, so it would use minimal bandwidth.

The problem with Leap is that to do anything remotely accurately for a moderate period of time takes an huge amount of effort moving your arm in mid-air with only visual feedback. It will be marginally worse than using a wii controller to type on a screen keyboard, as that includes tactile vibration feedback as you hover over keyboard letters. In the video it struggled at typing a single word on a keyboard.

The device looks great for toy/demo cases like google earth and fruit ninja. There aren't many normal use cases where you would need slow, low accuracy pointing or gestures from a distance that isn't already covered by wii, kinect or a laser pointer for example.

Remote control is already solved by remote controls with buttons and smartphones connected to a tv. The tactile feedback increases the speed of usage so much that no touchless device will ever surpass.

Predictive typing for single words is usually based on a locally stored dictionary, and for sentences it can be done with a model like prediction by partial matching. The device functionality is similar to a system used for speech recognition, in that better accuracy would only be gained by sending motion and text data to the cloud at the very least in the training stage and then aggregate and analyse data using a prediction model.

However, no-one seems to care that most voice recognition tools like Siri nowadays send and store data on servers, and nearly all those aren't OSS.