Re:A HTTP Proxy Server in 20 Lines of node.js Code
by silvestrov on Wednesday April 28, 2010 @08:15AM
(#1301335)
This is not a server in 20 lines of code, but server configuration in 20 lines (there is no parsing of the http protocols, but only calls into a lib that implements an http proxy).
The examples could be implemented in fewer lines of Apache configuration. I would have been nice with some examples that shows what node.js can do, which can't be done (easily) with Apache.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by TimothyFitz on Wednesday April 28, 2010 @12:19PM
(#1301908)
Not quite accurate. The code he wrote does the "proxy" logic while libraries underneath it are doing the work of parsing HTTP. He's definitely not doing "calls into a lib that implements an http proxy."
More to the point, what he has is a proxy in 20 lines of code which will let him make, say, a custom load balancing http proxy in 40 lines of code.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by simonw on Wednesday April 28, 2010 @09:15AM
(#1301451)
I think this does show something that can't be done easily with Apache: scripting decisions about what to proxy in JavaScript, in this case to implement auto-reloading of configuration files for IP filtering and regex-based host blacklisting.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by simonw on Wednesday April 28, 2010 @07:56AM
(#1301304)
If anyone's looking for a project, a reusable HTTP proxying library for Node.js with easy hooks for adding pre- and post- checks and transforms would be incredibly useful. Being able to implement API rate limiting and authentication as a Node.js reverse proxy that wraps around an existing web application would be even easier with a well-tested, fully featured proxy library.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by andrewvc on Wednesday April 28, 2010 @06:50PM
(#1303005)
EDIT: After looking at this http://nodejs.org/jsconf2010.pdf from sh1mmer's post, it looks like I need to be using a Buffer object to serve data over 64kb due to v8's garbage collector making a performance mess when using strings.
EDIT2: After looking at the code again, file reads come in as strings, so I don't actually see that Buffers would help, at least until the node file API starts passing around buffers instead of strings.
I can't speak to performance wrt to proxying, but I benchmarked its performance serving files recently[1].
Now, I didn't benchmark nginx, but as you can see, its pretty slow at that task, averaging ~10 MiB/s on fairly decent hardware, across a single gigabit switch.
One problem node has, is that it sucks dealing with strings containing non UTF-8 data:
From the node docs:
Node supports 3 string encodings. UTF-8 ('utf8'), ASCII ('ascii'), and Binary ('binary'). 'ascii' and 'binary' only look at the first 8 bits of the 16bit JavaScript string characters.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by fizx on Wednesday April 28, 2010 @12:55PM
(#1302007)
I benchmarked this a few weeks ago at around ~300req/s on my recent macbook, which was a little disappointing.
For kicks, here's my CoffeeScript/Node.js code I used. I did the benchmark with ab entirely on localhost. Normally that's not ideal, but it was enough to tell that this isn't as fast as I'd like.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by euroclydon on Wednesday April 28, 2010 @09:19AM
(#1301457)
Since this is near the top of the HN page, I'd like to ask a question about node.js:
What is it suited for? I read the about section of the node.js homepage, and it says that "Node's goal is to provide an easy way to build scalable network programs." Okay, that sounds cool, and I like Javascript, so I want to see if I can fit it into any projects I'm working on, but then I go on to read: "Almost no function in Node directly performs I/O." Okay again, single thread, no I/O, I can see why it's fast. But what if I need I/O, like from a DB? Does that mean I should not use node.js? What is a suitable method to store and transport data when using node.js? MemCache?
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by simonw on Wednesday April 28, 2010 @09:44AM
(#1301505)
Database adapters for Node are beginning to show up - there's an async PostgreSQL one and I think there's one using libdrizzle (which should work for MySQL as well). Since Node speaks HTTP fluently it's great for talking to things like CouchDB, and there's also DBSlayer http://code.nytimes.com/projects/dbslayer which provides an HTTP API for running MySQL SQL queries.
Node + Redis is an awesome combination.
Don't forget APIs - if your application simply hits a bunch of HTTP APIs from other providers (mashup style) Node is a great fit.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by gcampbell on Wednesday April 28, 2010 @09:52AM
(#1301527)
The important word in that sentence is "directly". There are plenty of ways to do I/O in node, but they generally involve event-based callbacks so that they don't block the thread.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by euroclydon on Wednesday April 28, 2010 @09:55AM
(#1301539)
Are there any special considerations for keeping the TCP connection open while the thread is waiting for the callback? I'm assuming no, since node.js doesn't seem to exist in order to facilitate long running requests.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by simonw on Wednesday April 28, 2010 @10:20AM
(#1301595)
Node is designed specifically TO facilitate long-running requests. There is only one thread (it's a single-threaded, evented system). Everything is done with callbacks, so if you have a long running I/O operation you'll just add a callback which completes the current HTTP response when the long-running I/O has finished. Node is quite happy to keep thousands of such callbacks around at once for massive concurrency.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by Sidnicious on Wednesday April 28, 2010 @10:21AM
(#1301599)
Since Node scripts are persistent (i.e. aren't created and destroyed for each request), it doesn't care how long or how many connections are open. It's ideal for long-running requests.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by lftl on Wednesday April 28, 2010 @10:11AM
(#1301580)
I don't really have a direct answer to your question, but the somewhat official demo for node.js is a chat server that uses http long polling to push messages out, so node can definitely handle that.
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by calcnerd256 on Wednesday April 28, 2010 @10:07AM
(#1301570)
I'm not sure, but I'd guess closing over it in the callback would work, as long as the TCP timeout is longer than the database response (even then, you could probably touch the stream every so often while you wait, perhaps via nextTick or something).
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by alunny on Wednesday April 28, 2010 @09:29AM
(#1301475)
The core of Node has very few functions that perform I/O, but there's no reason you can't use I/O in your own programs (in the original link, the network stream is read and written out, and also the blacklist file is read, synchronously). The readFileSync call is an anomaly in Node - typically you use evented I/O - open a stream and set listeners/callbacks for particular events.
There are modules developed for most of the major open source databases (I'm not sure what the status of the async MySQL driver is, but there's definitely something you can use).
Re:A HTTP Proxy Server in 20 Lines of node.js Code
by radicalmatt on Friday May 21, 2010 @01:13AM
(#1366456)
I suppose I should chime in and say that I am not exactly sure what Node.js's about page is talking about when it says 'I/O'. Could someone clue me in as to what exactly this means?
The examples could be implemented in fewer lines of Apache configuration. I would have been nice with some examples that shows what node.js can do, which can't be done (easily) with Apache.
More to the point, what he has is a proxy in 20 lines of code which will let him make, say, a custom load balancing http proxy in 40 lines of code.
EDIT2: After looking at the code again, file reads come in as strings, so I don't actually see that Buffers would help, at least until the node file API starts passing around buffers instead of strings.
I can't speak to performance wrt to proxying, but I benchmarked its performance serving files recently[1].
Now, I didn't benchmark nginx, but as you can see, its pretty slow at that task, averaging ~10 MiB/s on fairly decent hardware, across a single gigabit switch.
One problem node has, is that it sucks dealing with strings containing non UTF-8 data:
From the node docs:
Node supports 3 string encodings. UTF-8 ('utf8'), ASCII ('ascii'), and Binary ('binary'). 'ascii' and 'binary' only look at the first 8 bits of the 16bit JavaScript string characters.
1. http://blog.andrewvc.com/nodejs-can-really-scale
Summary: Node isn't quite as fast as NginX but it's close, and it's programable, and he's still tuning it.
For kicks, here's my CoffeeScript/Node.js code I used. I did the benchmark with ab entirely on localhost. Normally that's not ideal, but it was enough to tell that this isn't as fast as I'd like.
http://gist.github.com/382372
So your result is probably an order of magnitude off.
Edit: I feel like I tried that, and a couple other variations, and this was the fastest. But it's worth revisiting, and I'll definitely try it again.
What is it suited for? I read the about section of the node.js homepage, and it says that "Node's goal is to provide an easy way to build scalable network programs." Okay, that sounds cool, and I like Javascript, so I want to see if I can fit it into any projects I'm working on, but then I go on to read: "Almost no function in Node directly performs I/O." Okay again, single thread, no I/O, I can see why it's fast. But what if I need I/O, like from a DB? Does that mean I should not use node.js? What is a suitable method to store and transport data when using node.js? MemCache?
Node + Redis is an awesome combination.
Don't forget APIs - if your application simply hits a bunch of HTTP APIs from other providers (mashup style) Node is a great fit.
http://github.com/fictorial/redis-node-client
There are modules developed for most of the major open source databases (I'm not sure what the status of the async MySQL driver is, but there's definitely something you can use).