News for hackers. Stuff that matters.
Re:Wait.for: Sequential programming for Node.js (beta) by chrisohara on Thursday August 22, 2013 @11:33PM (#6261554)
Why not use harmony generators?

Generators are provided by V8, don't require an external fibers extension (i.e. hack), and will be available in the next node stable via `node --harmony-generators`.

Here's an example for those interested: https://github.com/visionmedia/co

Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Friday August 23, 2013 @12:18AM (#6261686)
node-fibers is under the hood, and works on the actual node stable. I hope ES6 generators -when stable- will provide similar tools, so wait.for will be based on ES6 generators.
Re:Wait.for: Sequential programming for Node.js (beta) by FooBarWidget on Friday August 23, 2013 @06:02AM (#6262565)
> What if... Fibers and WaitFor were part of node core?

> then you can deprecate almost half the functions at: http://nodejs.org/api/fs.html (clue: the Sync versions)

Not quite. I use the Sync versions not to avoid callbacks, but to ensure that nothing else happens while I'm doing the operation. For example, take a look at the Database.reload function from this project (a NoSQL database written in Node): https://github.com/phusion/zangetsu/blob/master/lib/zangetsu.... Reloading the database contents involves traversing multiple directories and reading multiple files. While reloading the database contents, I do not want the database to be modified. With asynchrony, the amount of different states that I have to be aware of rises so quickly that it's much, MUCH easier to use sync calls and tell the user "reloading is not concurrent, deal with it; at least it's correct". Wait.for does not solve this problem.

It seems people believe that you can't run into concurrency problems or race conditions because Node is single threaded. This is false. I even had to write my own locks (https://github.com/phusion/zangetsu/blob/master/lib/zangetsu...) because Node doesn't provide one.

Re:Wait.for: Sequential programming for Node.js (beta) by casual_slacker on Thursday August 22, 2013 @10:53PM (#6261432)
Cool idea. You should mention upfront that you're using Fibers under the hood, since it has important performance implications. Also, your Wait.for function only takes functions right now; but if you check for a .then function you can accept promises too.
Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Thursday August 22, 2013 @11:04PM (#6261457)
Thanks. I've put a link to node-fibers, 2nd line of readme. ¿are Fiber's performance problems solvable? ¿what if fibers were integrated in node core?
Re:Wait.for: Sequential programming for Node.js (beta) by AdrianRossouw on Friday August 23, 2013 @04:38AM (#6262370)
fibers will never be integrated into node core.

for what it's worth, I also refuse to use or depend on any code that makes use of node-fibers. I know i'm far from the only person who feels that way too.

Re:Wait.for: Sequential programming for Node.js (beta) by olegp on Friday August 23, 2013 @04:17AM (#6262319)
If all you want to do is make existing async methods synchronous, it's easier to modify the function prototoype like this: https://github.com/olegp/mongo-sync/blob/master/lib/mongo-sy... or just use Futures that come with fibers.

However, in practice you want a higher level of abstraction, since multiple async calls typically end up mapping to one sync call. This applies to streams in particular where you need to wait until they become readable.

I've written Common Node (https://github.com/olegp/common-node) to do just this and made it CommonJS and RingoJS compatible, which means the code written for Common Node runs on both V8 and the JVM. We've been using it in production at https://starthq.com for a while with great success. Due to some optimizations we put in place, Common Node actually performs faster than Node in some benchmarks.

Re:Wait.for: Sequential programming for Node.js (beta) by scriby on Friday August 23, 2013 @11:38AM (#6264240)
https://github.com/scriby/asyncblock

I'll just leave this here. It's been my pet project for the last couple years, and is a fully featured version of this concept.

There's also https://github.com/0ctave/node-sync, which is the same concept.

Re:Wait.for: Sequential programming for Node.js (beta) by zongitsrinzler on Friday August 23, 2013 @01:31AM (#6261886)
Correct me if I'm wrong but isn't a big part of async programming that it forces you to think about error handling. Working as a Java developer I have seen very lousy error handling mostly because try/catch lets you wrap everything and kill errors easily. Most of the node code I have seen handles errors much better thanks to it's async nature.
Re:Wait.for: Sequential programming for Node.js (beta) by gruseom on Friday August 23, 2013 @01:54AM (#6261927)
If you do it well, you can get very precise error handling, because errors are detected in the context where they first occur. However, to do it well you have to put a try/catch around every asynchronous operation, which makes nested callbacks even harder to read than they already are. Consequently people write code that doesn't handle all possible errors, and when one falls between the cracks, it gets caught only by the top-level default handler (assuming you've declared one), at which point the context is lost.

The other problem that comes up is the difficulty of getting a backtrace beyond the current callback. It can be hard to troubleshoot an error in a callback when you can't look back at how the async call got started. In such cases what you really want is a list of backtraces that capture all the relevant history. I seem to recall that people have come up with solutions for this, but can't remember what they are. Maybe someone else can comment?

Re:Wait.for: Sequential programming for Node.js (beta) by tg3 on Friday August 23, 2013 @03:13AM (#6262152)
Most Node.js I've seen doesn't make extensive use of Javascript's built-in exception handling, and instead relies on the convention of the first parameter of a callback being an Error object if one occurred (so no try/catch, which doesn't catch async errors anyway).

The stack trace is a pain for those operations, and any error that occurs as part of "normal" javascript operation (e.g. ReferenceError) has to be handled like a normal exception, which makes the style inconsistent and therefore a bit harder to manage.

I think that the latest or upcoming version of Node is presenting a solution to this in the form of domains, but I'm not too familiar with it.

Re:Wait.for: Sequential programming for Node.js (beta) by gruseom on Friday August 23, 2013 @03:55AM (#6262273)
The first-parameter convention only works for application errors and errors from libraries that also follow the convention. Everything else, including the JS runtime, throws exceptions whether we like it or not. So either you have to make sure that your code never triggers any of those—which is a pretty tall order—or put try/catches around everything, even if your own code doesn't throw any exceptions. That's more than a bit harder to manage—if you do it correctly, there are so many try-catch blocks that the rest of the code almost gets drowned out by them. In my observation, most people deal with this by sacrificing a certain amount of error safety in order to make the code more tractable.

It seems like the kind of problem that is better dealt with by a compiler.

Re:Wait.for: Sequential programming for Node.js (beta) by aaronyo on Friday August 23, 2013 @04:27AM (#6262340)
Promises help with this quite a lot. http://promises-aplus.github.io/promises-spec/ What 3.2.6.2 means is that any thrown exception is caught and turned into a rejected promise which will bubble up (across async boundaries) until it gets to a rejection handler.

I haven't had too much experience with Domains, but I believe they are generally used for courser grain error handling (because it's a bit verbose to attach them all over the place). E.g., to catch most errors (it's no guarantee what 3p libs do to your context) generated in fulfillment of a particular web request.

Re:Wait.for: Sequential programming for Node.js (beta) by gruseom on Friday August 23, 2013 @12:41PM (#6264668)
Can promises guarantee that condition with only library code? or do you still have to have try/catch in the application code, and if so where?

Also, do any of the promises libraries solve the backtrace problem, and if so how do they do it?

Re:Wait.for: Sequential programming for Node.js (beta) by tg3 on Friday August 23, 2013 @03:44PM (#6265871)
Unfortunately putting a try/catch around a 3p lib doesn't actually capture exceptions generated by those libs (unless they occur before any async operations, which I would say is the exception rather than the rule).

You're right that you have to manage exceptions, which is difficult. The biggest problem I've had with Node is the runtime exceptions that crash the entire process because they go uncaught. To deal with that, I have the server automatically restart on failure - far from an ideal scenario of error recovery (and what I think you're referring to when you talk about sacrificing error safety).

Re:Wait.for: Sequential programming for Node.js (beta) by zongitsrinzler on Friday August 23, 2013 @02:05AM (#6261958)
I think this is a good point, sometimes you don't need too precise error handling, then try/catch is very good.
Re:Wait.for: Sequential programming for Node.js (beta) by ilaksh on Friday August 23, 2013 @05:47AM (#6262536)
Please also take a look at ToffeeScript which is a brilliant and amazingly relatively unknown solution. https://github.com/jiangmiao/toffee-script
Re:Wait.for: Sequential programming for Node.js (beta) by SeanDav on Friday August 23, 2013 @06:37AM (#6262651)
Not dissing your comment, but I am not a believer in any "compile to" language because ultimately when things go wrong you are forced to deal with the underlying language in any case. Rather become an expert in the underlying language and learn to deal with its idiosyncrasies.
Re:Wait.for: Sequential programming for Node.js (beta) by ilaksh on Sunday August 25, 2013 @01:40AM (#6271051)
I'm not suggesting that people shouldn't learn JavaScript. Anyway all languages compile are abstractions of some lower level (usually multiple levels).
Re:Wait.for: Sequential programming for Node.js (beta) by tspike on Friday August 23, 2013 @12:24AM (#6261708)
Is this different from async.series? If so, in what way?
Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Friday August 23, 2013 @12:36AM (#6261743)
- syntax

- wait.for it's simpler

- wait.for uses try/catch

I don't know async.series deeply, but, ¿how do you use results from the first function on the second or third?

wait.for:

  try{
    data = wait.for(function1);
    result = wait.for(function2, data);    
  }
  catch(err){
    // error handling
  };
async.series:

  async.series([
    function1,
    function2
    ],
    // optional callback
    function(err, results){
      // results is now equal to ['one', 'two']
    });
async has a lot of functionality. You can use wait.for and async at the same time. They supply different functionality.

Example:

  results = wait.for(async.map,['file1','file2','file3'], fs.stat);
Re:Wait.for: Sequential programming for Node.js (beta) by htilford on Friday August 23, 2013 @12:54AM (#6261782)
> ¿how do you use results from the first function on the second or third?

waterfall

  async.waterfall([
    function (cb) {
      . . .
      cb(stuff);
    },
    function (stuff, cb) {
      . . .
Re:Wait.for: Sequential programming for Node.js (beta) by ibizaman on Friday August 23, 2013 @11:53AM (#6264347)
Wouldn't it be easier to just not use Node.js then ? :p
[deleted]
Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Thursday August 22, 2013 @11:22PM (#6261524)
A fiber is a particularly lightweight thread of execution. basically coroutines.

http://en.wikipedia.org/wiki/Fiber_(computer_science)

in node.js, fibers are... http://sstur.com/pres-fibers

Re:Wait.for: Sequential programming for Node.js (beta) by EGreg on Thursday August 22, 2013 @11:17PM (#6261500)
I don't get it,

    console.log(wait.for(fs.readfile,'/etc/passwd'));
won't that execute console.log immediately, whereas fs.readfile will return the data later? "I don't think it means what you think it means" http://www.youtube.com/watch?v=G2y8Sx4B2Sk
Re:Wait.for: Sequential programming for Node.js (beta) by qznc on Friday August 23, 2013 @02:22AM (#6262005)
A function executes once all its arguments are evaluated. console.log executes once its single argument wait.for(...) is evaluated. Hence: No, it will not execute immediately.
Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Thursday August 22, 2013 @11:27PM (#6261537)
wait.for, waits for fs.readfile to callback, and returns "data" from callback(err,data)

wait.for(fs.readfile,'/etc/passwd') == fs.readfileSync('/etc/passwd')

wait.for allows you to call any async function as if it were a sync function, without blocking node's event loop. It works inside a fiber. Check the examples.

Re:Wait.for: Sequential programming for Node.js (beta) by unnuun on Thursday August 22, 2013 @11:42PM (#6261578)
Right, but how the heck is this achieved? Could you show a line or two of example pseudo code that demonstrates the core of this functionality?
Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Thursday August 22, 2013 @11:56PM (#6261628)
-pseudo code-

  function wait.for ( asyncFn, params ) {

    //call the async fn, returns immediately
    asyncFn(params, function(err,data){
               //here the data is ready
               thisFiber.err = err;
               thisFiber.data = data;
               thisFiber.resume(); //resume execution (after yield())
    });

    //we're in a fiber, so we can yield() to node's event loop 
    thisFiber.yield(); //pause here, give back control to node's event loop
    
    // the next line will be executed when thisFiber.resume() is called
    if (thisFiber.err) throw thisFiber.err; // throw into the caller of wait.for
    return thisFiber.data; //ok
  }
Re:Wait.for: Sequential programming for Node.js (beta) by tg3 on Thursday August 22, 2013 @11:54PM (#6261618)
It uses Fibers (https://github.com/laverdet/node-fibers). I don't know much about it, but you have to wrap all your code in a call to `Fiber()`, which is written in C++.
Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Friday August 23, 2013 @12:10AM (#6261664)
Fiber() is under the hood, and I hope ES6 generators -when stable- will provide similar tools, so wait.for could be based on ES6 generators.

You don't have to wrap all your code in a call to `Fiber()`

full classic node-server sample:

    var http=require('http'), wait=require('wait.for');

    http.createServer(function(req, res) {
      wait.launchFiber(handleRequest,req, res); //run in a Fiber, keep node spinning
    }).listen(8080);


   //in a fiber
   function handleRequest(req, res) {
     try {
       res.writeHead(200, {'Content-Type': 'text/html'});
       res.end(
       "<h1>Demo page</h1>" 
       + "<p>"+ wait.for(fs.readFile,'/etc/somefile') +"</p>"
       + "<p>"+ wait.for(fs.readFile,'/etc/otherfile') +"</p>");
     }
     catch(err) {
      res.end('error '+e.message); 
     }
   }
Re:Wait.for: Sequential programming for Node.js (beta) by tg3 on Friday August 23, 2013 @03:08AM (#6262134)
in your example below, you do have to wrap it in a call to `Fiber()`, although you've disguised it as `launchFiber`, which you pass `handleRequest` to (I would call that wrapping `handleRequest`.

That is probably a worthwhile tradeoff, but the first time I read through it, that mechanic wasn't obvious to me.

Re:Wait.for: Sequential programming for Node.js (beta) by lucio on Friday August 23, 2013 @04:05AM (#6262296)
It's not 'disguised', it's abstracted. launchFiber does what it's name implies: launchs a fiber. How it is achieved 'internally', it's irrelevant as long the abstraction is not leaky.

Today launchFiber is calling the fn inside a Fiber(), tomorrow could be launching the fiber by means of a generator.

The idea was to create a very simple abstraction to avoid callback hell, adding low or no complexity, and keeping the flexibility to use all the existing async fns with a callback or just "wait" for the results.

Re:Wait.for: Sequential programming for Node.js (beta) by anko on Friday August 23, 2013 @12:02AM (#6261647)
fibers are lightweight threads, employing a sort of non-preemptive multitasking paradigm. it basically spawns a 'thread' that will be resumed only when the callback has returned, which returns the data passed to the callback.

the code is here: https://github.com/luciotato/waitfor/blob/master/waitfor.js, but it does require fibers.

Re:Wait.for: Sequential programming for Node.js (beta) by cypher543 on Friday August 23, 2013 @12:17AM (#6261684)
The code is open source and only 72 lines long (with comments and whitespace).