I hate to break it to you, but unless you store your home folder on a separate encrypted partition (which would be really inconvenient), this won't work.

FileVault 2 encrypts your entire boot volume, so there's no separate encryption of your home folder going on. If you give someone any of the passwords that unlock the boot volume, they could get all of the data off it (by using target disk mode or booting to single user mode, etc.)