News for hackers. Stuff that matters.

Parallel Python or Ruby?

If you were to add some parallel language constructs to Python or Ruby or <insert your favourite language>, what would you add? Lets assume you are only allowed to add 1 or 2 constructs so you dont go changing the whole language upside-down. Also assume that you are not allowed to take something out of the language only add to it. Please describe in detail with actual examples if possible. Dont just say "I will add actors" at a very high level.
( 20 comments | 6 points | on HN )
Re:Parallel Python or Ruby? by kilowatt on Saturday June 21, 2008 @09:32PM (#223979)
Keep in mind that Python 2.6 already has the new multiprocessing package, which presents a simple interface for multiprocess computation.

I'd like to poke around with Python's compiler module and look at the AST--you could enforce that a certain class of functions have no side effects, and then spawn multiple processes to execute them semi magically.

Futures like the ones in IO (http://www.iolanguage.com/scm/git/checkout/Io/docs/IoGuide.h...) always struck me as a pretty cool way to abstract away synchronization problems. Using the multiprocess stuff as a background would be fun.

Re:Parallel Python or Ruby? by randomhack on Saturday June 21, 2008 @10:34PM (#224006)
Hmm is poking around the AST necessary? What about looking at the bytecode instead?
Re:Parallel Python or Ruby? by dhotson on Saturday June 21, 2008 @09:38PM (#223980)
One change to ruby that would be useful, would be to make Enumerable.collect run in parallel.

There's probably a whole bunch of other Enumerable methods that could be made to run in parallel as well.

Re:Parallel Python or Ruby? by fizx on Saturday June 21, 2008 @11:54PM (#224033)
gem install facets require "rubygems" require "facets/thread" %w[a b c].threaded_map{|str| puts str }

It's multicore if you're using jruby.

Re:Parallel Python or Ruby? by jrockway on Saturday June 21, 2008 @10:09PM (#223994)
If you use Coro with Perl, you can say something like:

  async {
    do_something;
    cede;
    do_something;
  }

  do_something_else;
  cede;

See http://search.cpan.org/dist/Coro/Coro.pm for more information.

[Edit: If you want to see a continuation-based web framework built with Coro, look at Continuity: http://search.cpan.org/~awwaiid/Continuity-0.994/lib/Continu...

It's very cool.]

Re:Parallel Python or Ruby? by Kaizyn on Sunday June 22, 2008 @12:29AM (#224051)
Hello randomhack,

Have you heard of Stackless Python? That might be what you are looking for. http://www.stackless.com/

Re:Parallel Python or Ruby? by randomhack on Sunday June 22, 2008 @12:48AM (#224059)
I have read a bit about stackless. Neat implementation but it still suffers from GIL? You can launch thousands of microthreads but apparently they still run on single core?
Re:Parallel Python or Ruby? by tzury on Sunday June 22, 2008 @01:03AM (#224063)
Do we have here a solution looking for a problem?
Re:Parallel Python or Ruby? by tptacek on Sunday June 22, 2008 @01:34AM (#224078)
Very yes.
Re:Parallel Python or Ruby? by throttle on Saturday June 21, 2008 @09:40PM (#223982)
Haven't you heard about the egregious buffer overflow vulnerabilities found recently in Ruby? Not a good idea to use it for anything serious in light of those, IMO.
Re:Parallel Python or Ruby? by jrockway on Saturday June 21, 2008 @10:09PM (#223993)
I'm not a Ruby fan, but wouldn't finding those buffer overflows make it more secure? Now that they've been found, they're fixed. (Sure, it could be indicative of overall careless design, but I don't think that's the case here. A few silly mistakes, now no longer a problem.)

Finally, Ruby has more than one implementation. Finding a hole in one implementation says nothing about the language overall.

Re:Parallel Python or Ruby? by tptacek on Sunday June 22, 2008 @01:38AM (#224083)
Yes. Everyone else: find a large C codebase that has shipped without an integer overflow. Go ahead. I'm waiting.
Re:Parallel Python or Ruby? by tptacek on Sunday June 22, 2008 @03:57AM (#224129)
Want some help? Don't bother with qmail. Daniel J. Bernstein managed to ship a version of qmail with an integer overflow.

(That's the only vulnerability ever found in qmail, and it wasn't exploitable --- Postfix has had much worse).

Re:Parallel Python or Ruby? by jrockway on Sunday June 22, 2008 @11:48PM (#224710)
Too bad the original qmail is nearly useless these days. (I had a class with DJB in college, and used qmail for quite a while. Eventually I got overrun with spam and switched to something that would reject messages immediately after virus/spam scanning them.)

The point is that writing simple software in C is possible. But writing complex software is very very hard.

Re:Parallel Python or Ruby? by tptacek on Monday June 23, 2008 @12:13AM (#224741)
Every mail server I've ever run has been qmail, including my current company's current mail server, which backs a relatively popular blog and a pretty healthy number of press hits with @matasano.com mail addresses, along with several decently active open mailing lists, and I simply don't have any of these problems you're talking about.

If you took a class with Bernstein, you might be familiar with the ISP whose entire mail operation, including customer POP and virtual hosting, I ran on qmail --- that'd be EnterAct. So yeah, I'm not buying your "qmail is too simple to matter" argument. If you want to chase it down, I'll be happy to show you where the last few Sendmail vulnerabilities were found. Clue: not in the crazy Sendmail features that qmail lacks.

But that's besides the point. You didn't answer my question.

Name a piece of code of comparable complexity to qmail that's never had an integer overflow. Here's another hint: Perl isn't one of them.

Re:Parallel Python or Ruby? by throttle on Saturday June 21, 2008 @10:16PM (#223998)
The mistakes are so amateurish that it should raise questions about the rest of the codebase (i.e. the defects which are yet to be found)...
Re:Parallel Python or Ruby? by j2d2 on Saturday June 21, 2008 @11:40PM (#224026)
What projects have you done?
Re:Parallel Python or Ruby? by jrockway on Sunday June 22, 2008 @12:31AM (#224055)
should raise questions about the rest of the codebase (i.e. the defects which are yet to be found)

It has.

Re:Parallel Python or Ruby? by tptacek on Sunday June 22, 2008 @01:37AM (#224081)
By who? Cite someone who matters.
Re:Parallel Python or Ruby? by tptacek on Sunday June 22, 2008 @01:36AM (#224080)
Haven't you heard about JF's talk at PH-Neutral and the memory corruption vulns they found in Python and Perl? Not a good idea to use those for anything serious either. I guess we're all stuck with Erlang.