That button exists, it's blue and written "Following" and turns red with the label "Unfollow" when you hover.

And yes, you can configure twitter + using tweetdeck to only see tweets in chronological order and only from people you follow.