News for hackers. Stuff that matters.
Re:Web Scraping with Node.js and Chimera by kanzure on Wednesday January 02, 2013 @08:31PM (#5000561)
Great project. The biggest question for me when I'm using phantomjs is why phantomjs is trying to replicate nodejs infrastructure. For example, phantomjs has an HTTP server feature for processing incoming requests. This doesn't make sense to me because a browser shouldn't be a server. If you need to get information out of the worker, you should POST it somewhere. The proclivity of phantomjs users to prefer stdout is astounding. It's definitely the #1 question or issue that I get fielded in #phantomjs on freenode.

For example, for POSTing and reading from redis/resque I wrote this (proof of concept, not what's in production):

https://gist.github.com/000037f472b72d9490a6

A few thoughts..

    > There are similar "glues" like phantomjs-node that integrate phantomjs by
    > spawning a process, and processing the stdout stream, but it is limited by 
    > what can be done via the command line of phantomjs. If you really want direct
    > api access to the browser, the best way is via direct integration.
This seems like a lot of overhead on top of a phantomjs (or even just a generic webkit) worker. Substack's approach was to just put a proxy in front of a browser that injects a <script> tag into the page to boss the browser around:

https://github.com/substack/schoolbus

Supposedly the actual browser client shouldn't matter, as long as your fleet of workers are up and running. I bet chimera's approach will end up with more access to npm modules in the long run compared to phantomjs.

Also, the link wasn't in the article: https://github.com/deanmao/node-chimera

For the python equivalent of this project, there's https://github.com/kanzure/pyphantomjs

Re:Web Scraping with Node.js and Chimera by niggler on Wednesday January 02, 2013 @08:31PM (#5000562)
Did we really reach the point where demonstration code can be presented in coffeescript without an equivalent javascript demo?
Re:Web Scraping with Node.js and Chimera by chc on Wednesday January 02, 2013 @08:53PM (#5000643)
People have been doing that for a surprisingly long time. I think it was encouraged by the fact that CoffeeScript is immediately intelligible to a lot of JavaScript coders, and authors reckon that anyone who has trouble can just compile it and get fairly idiomatic JavaScript.

I actually wrote a bookmarklet a while back that would look for CoffeeScript snippets on a page and translate them for people who find it troublesome, but didn't end up doing much with it because I didn't feel like there was much interest.

Re:Web Scraping with Node.js and Chimera by niggler on Wednesday January 02, 2013 @09:05PM (#5000681)
criticisms of coffeescript aside (e.g. interstitial whitespace sensitivity), the code can't directly be applied in node (as far as i can tell, you can't tell node to run a coffeescript file directly using the `node <script>` syntax -- you have to use a framework or compile to js).
Re:Web Scraping with Node.js and Chimera by bdcravens on Wednesday January 02, 2013 @11:54PM (#5001086)
Check out the PeepCode "Full Stack node.js" screencasts - it shows building a full app using CoffeeScript.

https://peepcode.com/products/full-stack-nodejs-i

Re:Web Scraping with Node.js and Chimera by rcsorensen on Wednesday January 02, 2013 @10:05PM (#5000840)
It can after you install coffee-script.

  $ echo 'process.stdout.write "Now with CS level #{1 + 2}!"' > foo.coffee
  $ coffee foo.coffee
  Now with CS level 3!
  $
Re:Web Scraping with Node.js and Chimera by chc on Wednesday January 02, 2013 @09:59PM (#5000830)
If you don't want to compile, assuming you have CoffeeScript installed, you can just run it with coffee instead of node and everything else works like normal.
Re:Web Scraping with Node.js and Chimera by funkiee on Wednesday January 02, 2013 @09:10PM (#5000695)
How is that really different toolchain-wise from having to compile source code in a language such as Java?
Re:Web Scraping with Node.js and Chimera by niggler on Wednesday January 02, 2013 @09:21PM (#5000733)
remember: java is to javascript as pain is to painting.

it's not an apt comparison. You can directly write javascript and run it in your browser or in node (or test it out interactively in the node REPL).

I like to play with modules in node interactively (when relevant) because its easier to see what's going on and much easier to iterate (esp. in conjunction with the .load REPL command)

Re:Web Scraping with Node.js and Chimera by NuZZ on Wednesday January 02, 2013 @10:09PM (#5000848)
otf compiling is pretty simple regardless.

index.js:

    require('coffee-script')
    module.exports = require('app.coffee')
Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @12:24AM (#5001156)
The github project does have examples in standard javascript as well:

https://github.com/deanmao/node-chimera/blob/master/example....

Re:Web Scraping with Node.js and Chimera by fruchtose on Thursday January 03, 2013 @03:01AM (#5001411)
Great work! This might even have potential for browser-based testing, since mocha-phantomjs runs from an executable; I'd prefer a code-based solution like Chimera integrate with Mocha.
Re:Web Scraping with Node.js and Chimera by lancefisher on Wednesday January 02, 2013 @11:11PM (#5000980)
This is a great idea! phantomjs-node works okay, but it is suck a hack. A nifty hack, but still. https://github.com/sgentle/phantomjs-node#how-does-it-work

If you want to parse the DOM for the internet at large, you need a real browser. There are simply too many sites with really bad HTML to be parsed reliably with anything else.

Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @12:37AM (#5001182)
If you need a great html parser, I also wrote a library for one of those: https://github.com/deanmao/node-hubbub

It's merely an integration library for the netsurf browser. It's the html parser for a real browser. I considered using the parser from other browsers like firefox or webkit, but netsurf had the fewest external dependencies.

Re:Web Scraping with Node.js and Chimera by salmanapk on Thursday January 03, 2013 @01:43AM (#5001284)
Just in case this saves someone's time:

phantomjs-node uses ridiculous hacks. Not being rude but I think the author was drunk when writing that. I ended up wasting quite some time when trying to customize it.

You do NOT need to use any hacks whatsoever to run dnode on phantom, just browserify(https://github.com/substack/node-browserify) the shoe and dnode modules and use the example code at https://github.com/substack/dnode .

Re:Web Scraping with Node.js and Chimera by seanlinehan on Wednesday January 02, 2013 @07:58PM (#5000425)
This looks really great. I would love to see a bit more of an in-depth example in a follow-up post!

Is more documentation to come?

Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @12:22AM (#5001151)
Sure, I can make more. This post caught me by surprise. It's odd to see your own blog post from months ago on HN.

I sorta gave up on the project seeing as nobody other than myself used it. (I have zero watchers on github)

Re:Web Scraping with Node.js and Chimera by detst on Thursday January 03, 2013 @02:36AM (#5001378)
A README is the absolute bare minimum for a GitHub project. Even a lack of code can get some interest in an idea. Someone landing on the GitHub page will reflexively look for the back button if they haven't seen the blog post.

Great work!

Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @11:48AM (#5003180)
I've added a readme now. Will add more soon.
Re:Web Scraping with Node.js and Chimera by Trindaz on Sunday January 13, 2013 @12:02AM (#5050035)
Has anyone actually gotten this working? I've tried installing on Mac OS X and Ubuntu, both with various problems. The precompiled binaries don't work, the qt build scripts fail, etc. etc.
Re:Web Scraping with Node.js and Chimera by rco8786 on Wednesday January 02, 2013 @08:07PM (#5000461)
Is there more documentation(or source) available somewhere?
Re:Web Scraping with Node.js and Chimera by KaoruAoiShiho on Wednesday January 02, 2013 @08:34PM (#5000575)
It's on github, but at 6 months since last commit and no documentation...
Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @12:41AM (#5001187)
I can make more. After about 3 months with no github watchers, I figured the only person reading the docs would be myself.
Re:Web Scraping with Node.js and Chimera by KaoruAoiShiho on Thursday January 03, 2013 @01:37PM (#5003784)
Thanks for adding docs, this should be a better alternative than phantom or jsdom.
Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @11:48AM (#5003175)
I've added a readme now... will add more documentation soon.
Re:Web Scraping with Node.js and Chimera by mcantelon on Wednesday January 02, 2013 @07:53PM (#5000411)
Re:Web Scraping with Node.js and Chimera by CoffeeDregs on Wednesday January 02, 2013 @08:26PM (#5000533)
Sorta. The joy of PhantomJS (and of Chimera) is that they use a real browser to run the JS/CSS/HTML/whatever. No simulated DOM; no simulated cookies; etc. Just a real [headless] WebKit browser with all of its quirks and tricks. You can even take screenshots of a real, rendered webpage (which is great for debugging).
Re:Web Scraping with Node.js and Chimera by mcantelon on Wednesday January 02, 2013 @11:45PM (#5001064)
>You can even take screenshots of a real, rendered webpage (which is great for debugging).

That is pretty cool.

Re:Web Scraping with Node.js and Chimera by nodemaker on Wednesday January 02, 2013 @11:06PM (#5000971)
Re:Web Scraping with Node.js and Chimera by goldfeld on Wednesday January 02, 2013 @09:26PM (#5000748)
How does it differ from ZombieJS?
Re:Web Scraping with Node.js and Chimera by foxbarrington on Wednesday January 02, 2013 @09:58PM (#5000828)
Zombie doesn't use a "real" browser, but a simulated DOM Window using jsdom.
Re:Web Scraping with Node.js and Chimera by booz on Thursday January 03, 2013 @12:36AM (#5001181)
wow this is exactly what I need, thanks!