Responsive Design implemented incorrectly would decrease usability, but done correctly it almost certainly enhances it. The key premise of RWD is that the same functionality is available at all reasonable device widths (> 320px). This does not mean it needs to look the same, but you must be able to accomplish the same interactions on a smartphone browser as a desktop browser.

The goal of the Open Web is to provide a OS platform-agnostic foundation. The mistakes learned from building specifically towards IE6 and now building specifically towards WebKit in the mobile space have taught us that the platform-specific road is the road to narrowing your audience and higher maintenance costs down the road.

Sure, but you're sort of begging the question. It's very hard to do this perfectly correctly and in some cases I'm not sure you should try. A feature that's designed for a user with a mouse and a big monitor may be terrible to use on a phone even if you hack and squeeze and strip it to fit the screen.

I certainly agree that, done correctly, RWD enhances usability on all platforms. But the cost of doing RWD "correctly" (device testing and experience optimisation across a wide range of devices) can be prohibitively expensive when developing anything more than a landing page.

Any site that requires extensive user interaction (i.e. contains custom form controls, overlays etc.) requires that usage patterns be tailored to the context in which they appear. So at a minimum, you're going to want to change your interface based on whether the user is using a finger or a cursor. Screen size, browser capabilities etc. are all additional factors that a careful implementation needs to be aware of and adapt to.

Swiping gestures are supported on most modern phones today. And when they're not, you can gracefully fall back to touch events. And when those aren't supported, you could gracefully fall back to a list, rather than a slideshow. There are methods by which everybody gets to play along, and as long as those are implemented properly, nobody is left out.

You can do both.