News for hackers. Stuff that matters.
Re:EventMachine: scalable non-blocking i/o in ruby by tptacek on Monday March 15, 2010 @03:53PM (#1193911)
As luck would have it, I'm sitting here writing a fuzzer for a trading protocol in EventMachine as we speak.

Ruby and async network programming go together like gin and vermouth. It's one of the reasons I'm so happy with Ruby's block/lambda support, and so irritated with Python's: many of the basic programming idioms for evented network code have direct support in the Ruby language.

I've done a fair bit of Twisted Python programming in the past, and I find EventMachine to be far more intuitive and flexible. On the other hand, EventMachine will crash on me once in a blue moon, and it swallows exceptions, which is extremely frustrating. They're both great packages.

The big problem that still needs solving for EventMachine is evented adapters for an ORM (any ORM). You've always been able to talk to MySQL over an evented socket, but that's clumsy compared to DataMapper.

I am still totally not sold that Ruby threads are tenable, so a lot of the "advanced" stuff in EventMachine does nothing for me. It's great even without it.

Re:EventMachine: scalable non-blocking i/o in ruby by jrockway on Monday March 15, 2010 @04:07PM (#1193946)
I am still totally not sold that Ruby threads are tenable, so a lot of the "advanced" stuff in EventMachine does nothing for me.

FWIW, threads based on an event backend are a solved problem in Perl. The Coro library is what EM is trying to be; they should really take a look. ("Oh but it's PERL!!!!" Yeah, and it actually works.)

Re:EventMachine: scalable non-blocking i/o in ruby by sreque on Monday March 15, 2010 @05:28PM (#1194175)
I'll admit to being a major Perl hater. However, this Coro library looks great! Perl threads are horrible pieces of trash, and the Coro library author not only admits this but does something about it by implementing threads the way nature intended them to be. If I'm ever forced to code in Perl again and need or want threads, I will turn to this library first.

The way the Coro author implements Perl threads, by the way, sounds similar to how MacRuby and Rubinius are attempting to remove the GIL in ruby by copying interpreter state so that multiple OS threads can run simultaneously.

edit: after reading further, it turns out these threads are cooperative in that only one can run at a time. Also, it looks like they act more like Ruby fibers in that one must give up control explicitly or make a blocking call that implicitly gives up control. I'd rather the core scheduler be able to preempt threads directly. What if one of the threads takes a long time or ends up in an infinite loop?

"This module collection manages continuations in general, most often in the form of cooperative threads (also called coros, or simply "coro" in the documentation). They are similar to kernel threads but don't (in general) run in parallel at the same time even on SMP machines"

Re:EventMachine: scalable non-blocking i/o in ruby by jrockway on Monday March 15, 2010 @06:22PM (#1194304)
They can be automatically preempted if you desire, see: http://github.com/nothingmuch/coro/commit/54902809d8d3ad054d...

Usually people dismiss Coro because they want to take advantage of multiple cores without running multiple processes. But the reality is that you'll see a maximum 8x speedup if you run 8 OS threads, but if you rewrite your critical section in Haskell, you'll see a 50x speedup. So really , Coros are exactly the right abstraction for "scripting languages". IO performance is great, and expressiveness is great. Speed can be gained in other ways.

Re:EventMachine: scalable non-blocking i/o in ruby by sreque on Monday March 15, 2010 @08:39PM (#1194581)
I agree entirely about not needing threads in a scripting language like perl for CPU performance reasons, and being able to configure the threads to be scheduled automatically is a big win in my book. Thanks for letting me know about the library!
Re:EventMachine: scalable non-blocking i/o in ruby by silentbicycle on Monday March 15, 2010 @07:33PM (#1194430)
How does the Perl Coro library compare to coroutines in Lua or Scheme? (It's been almost a decade (!) since I used Perl really actively, and it seems to have changed quite a bit.)
Re:EventMachine: scalable non-blocking i/o in ruby by tptacek on Monday March 15, 2010 @04:14PM (#1193966)
Maybe, but part of the value of async programming is never having to thread, so none of this really lights me up.
Re:EventMachine: scalable non-blocking i/o in ruby by jrockway on Monday March 15, 2010 @06:23PM (#1194308)
Coro basically allows you to write:

   foo();
   bar();
instead of:

   foo( cb => sub { bar() } );
 
Behind the scenes, it's basically the same thing. It's cooperative so your code does not get preempted where you don't want it to be; the locking is implicit.

But you can switch to preemption and explicit locking if you desire. At that point, IMO, it's time to ditch Perl / Python / Ruby and use a language with a sane STM implementation, because locks suck.

Re:EventMachine: scalable non-blocking i/o in ruby by BrianHammond on Monday March 15, 2010 @05:01PM (#1194108)
Honest question: is there a purely event-driven I/O framework (no threads anywhere)?

The one that a lot of people have gotten excited about recently is Node.js and that uses threads for file I/O (libeio).

Re:EventMachine: scalable non-blocking i/o in ruby by jrockway on Monday March 15, 2010 @06:49PM (#1194361)
What you you think a "thread" is? A thread of execution? An OS thread? A lightweight thread?

Without defining thread, your question is pretty close to meaningless.

Re:EventMachine: scalable non-blocking i/o in ruby by WALoeIII on Monday March 15, 2010 @06:15PM (#1194290)
http://github.com/lifo/cramp

It works best in Thin (EventMachine) or Rainbows! (Fibers).

If you are doing I/O you should use NeverBlock (http://www.espace.com.eg/neverblock/) they've even implemented a MySQL driver, the best part is they're 'drop-in' so you don't have to learn a new API, they just hide the async calls from you.

Fibers I think are closer to Erlang style than the event loop, and much more straight forward to program.

Re:EventMachine: scalable non-blocking i/o in ruby by jemfinch on Monday March 15, 2010 @05:35PM (#1194195)
> Honest question: is there a purely event-driven I/O framework (no threads anywhere)?

You might look into a continuation-based framework. With continuations, you can do purely event-driven IO written in a direct style: the framework can grab the "callback" itself by snagging the continuation of its slow IO functions.

You still run into the same problems with blocking (non-IO, or non-framework) code taking up more than its fair share of time, however.

Re:EventMachine: scalable non-blocking i/o in ruby by tptacek on Monday March 15, 2010 @05:05PM (#1194118)
EventMachine doesn't thread unless you ask it to. I use EventMachine in Ruby and libevent in C with nary a mutex in sight.
Re:EventMachine: scalable non-blocking i/o in ruby by BrianHammond on Monday March 15, 2010 @06:01PM (#1194261)
I see. My (wrong) understanding was that EM used a thread pool to do file I/O like Node.js. It appears to add the file descriptor to the event loop's watch list.