Keep pluging away, it's way worth it :).
Rather than use the upgrade gem I started a new project, copied/converted what I could, and then let my tests tell me what was missing. First the models, then controllers, and finally the integration tests.
Without automated testing you're fucked of course, but as long as you have a decent safety net in place you'll get there.
I think it's best to think of Rails 3 as a separate project that happens to share some code with Rails 2.
I hope it's worth it in the end...