I dunno whether it would be considered hacking, fraud, or plain old theft, but if you hadn't returned the BTC, this would be a clear example of exploiting a bug for financial gain. if nothing else, the operator should have some legal recourse to get their money back. if it can be shown that you did it deliberately, I don't think it would be unreasonable to consider it a criminal offense.

of course, this is not quite the same as misunderstanding some subtleties in an explicit contract. for example, the mere fact that I did not understand an auto-renewal clause in a lease does not get me off the hook for payment. although I might have some recourse if the lease stipulates something absurd (eg, notice to vacate must be hand-delivered to office in antarctica).

crypto does not necessarily mean you get to do a complete end run around the laws of sovereign nations. what it might mean, if designed well, is that you don't have to depend on those nations enforcing things.