In my experience, Ruby and number crunching just don't match. Python's not especially fast to begin with, but I've definitely found it to be faster than Ruby. Though I probably wouldn't choose Python for number crunching either. If I were extracting some time-sensitive algorithm, I'd most likely go straight to something like Clojure.

And I must stress how much I love Ruby. It's my go-to language for the web/small scripts.