News for hackers. Stuff that matters.
Re:Ask HN: How does Mint.com work? by Sukotto on Thursday July 22, 2010 @10:11AM (#1538130)
Would you please share some code samples or favorite blog posts on how you're using phpQuery as part of a scrape app? What set of tools/libraries are you using?

I'd really like to hear more about the current state of the art (without you telling any company secrets). I have quite a lot of experience scraping utility and government websites (no javascript) in perl + LWP... but I'm getting a little tired of perl and am looking to give a new toolset a try. Perferably one that can handle a broader range of modern websites.

Re:Ask HN: How does Mint.com work? by encoderer on Thursday July 22, 2010 @04:37PM (#1539479)
I will throw you 1 bone, but past that, I'm careful because my work here is not my own, it's my employers.

libxml_use_internal_errors(true);

This little function call is the secret to solving what seem at first to be intractable memory-leaks. The trouble is the scraper uses libxml, and libxml issues a notice/warning every time the HTML is malformed. Without this call, those errs will bubble up to the PHP error handler and it'll murder performance and memory usage.

One more i suppose...

If you scrape inside a loop (and unless you're using a distributed job queue,if you're scraping more than one URL at a time you almost certainly are), missing a unloadDocument() call is going to cost you each time you iterate. The objects it creates, IIRC, have some circular dependency issues and if you don't explicitly unloadDocument you'll run into trouble. (Suppose it should be OK tho if you've enabled that GC feature in PHP 5.3)

And, generally, a tip... sometimes it's tempting to write a simple regex instead of a chain like, say, pq($this->node->find('a')->get(0))->attr('href')

Write the chain. Regex is just too brittle.

Re:Ask HN: How does Mint.com work? by Sukotto on Friday July 23, 2010 @09:06AM (#1541036)
Thanks for the tips. You most likely saved me a bunch of headache and I appreciate it.

I'm pretty intrigued by a library that can apparently handle ajax/json updates and content creation. Heh... I thought PHP was only for page generation and had no idea you could purpose it for something like webscraping.

So it should be fun playing with it.

My email's in my profile if you ever want to talk shop.

Re:Ask HN: How does Mint.com work? by justliving on Thursday July 22, 2010 @02:18PM (#1539037)
I have been using quite a bit of python and beautifulsoup.

http://www.crummy.com/software/BeautifulSoup/

Re:Ask HN: How does Mint.com work? by asnyder on Thursday July 22, 2010 @10:23AM (#1538179)
php|architect's recently released book entirely on this topic: http://www.phparch.com/books/phparchitects-guide-to-web-scra...
Re:Ask HN: How does Mint.com work? by l4u on Thursday July 22, 2010 @11:22AM (#1538393)
I have been using spynner, written in python. It uses pyQtWebkit, and jQuery. Use the trunk version at http://code.google.com/p/spynner/