I'm sure I'll be corrected if I'm wrong, but my understanding is that public key cryptography is very processor intensive. Because of this, instead of using your private key to sign every bit of communication, you use the public/private key crypto to negotiate a shared secret for a different algorithm, such as AES.

So it still comes down to having a shared secret. Public/private keys just change the way that secret is shared.

Going back to the issue at hand, the question is where did you get the original public key from. This has always been the "problem" in public key systems. You have to trust the public key that you get in the first place.

It's more than just an issue of efficiency. Encrypting/signing nonrandom text with RSA exposes you to all kinds of nasty attacks. For example, if you directly sign a message whose blocks are composed of prime numbers (rather than hashing the message and signing the digest), then an attacker can afterward forge your signature on any message composed of a sequence of blocks which are known powers of those primes or their mod-N inverses.