Not only might it be more fun, but you'd build a better intuition about what to expect from it (or even how it works).
It's like how people often make fewer mistakes with slide-rules or abacuses than pocket calculators, since the former two require you to check your answer against your intuition, while the latter just tells you "the answer" to your (potentially ill-formed) query. Trusting a black box can be quite dangerous.
I threw 'real' in there to cover that. Maybe that's not fair though ;)
More seriously though: if the oracle were perfect, you should have to do almost nothing (i.e. the relative time savings would be near 100%; e.g. creating pizza: the oracle, you just tell: "get me pizza", whereas with the bike, you have to bike there—which I admit may be fun)—so as long as you have things you enjoy doing more in your non-productive free time, then it shouldn't be an issue. (Not that I think that's necessarily likely...)
The bicycle might be more fun to ride