It refers to the cat o' nine tails, which was used to punish sailors in the Royal Navy and was, supposedly, kept in a special bag.
It's also where "not enough room to swing a cat" is said to come from.
http://upload.wikimedia.org/wikipedia/commons/0/09/Cat-o%27-...
.jpg){kind=link}
The nautical origin of the 'cat out of the bag' definitely refers to taking the cat o'9 tails out of the bag it was kept in. Whether that, in itself, was linked to hiding a cat in a bag to pass off in a market isn't clear. As always, there is dispute about the origins of terms.
This seems to correlate to a Hebrew expression "buying a cat in a sack", meaning taking something without confirming its quality.
Indeed, same legend is presented as the source for this expression as well.
Could you explain 'letting the cat out of the bag' in this context? I'm asking because I've read a different legend: Apparently on markets a few centuries ago, young pigs were frequently sold in bags. Sometimes people tried to sell captured cats as farrows in the hope the buyer would not open the bags before the deal was made and the seller far enough away. IIRC they chose cats because they sounded similar to farrows. Lacking farrows I cannot confirm that :)