Sorry, but you can kick a machine. You can not kick an algorithm, exactly like can you can not kick any kind of mathematical construction.
Now consider a physical general-purpose computer with your program loaded into it. Can you really argue that this is fundamentally a different kind of object than the special-purpose machine? They behave, let's suppose, identically.
Okay, maybe you would say, the program considered abstractly is still an object I can't kick, so it's not patentable; the only thing that's patentable is a physical computer with the program loaded into it. But that doesn't help you, because no one can actually use the program without loading it into a computer anyway. So if you get a patent on a computer with your program loaded, that has the same practical effect as a patent on the program itself. No?
Algorithms are mathematical structures, and they're also machines. Both of these claims are true in the plainest, most common meanings of the words possible. An attempt to create a distinction between classes of algorithms has to be based on something. Again: what could that be?
EDITED to add: just so it's clear where I'm coming from -- I think algorithms are clearly patentable subject matter. The problem we have with software patents is that the obviousness bar is far, far too low.