In theory, a Federal Reserve Note should be redeemable in United States Notes or in coins stamped by the U.S. Mint, but thanks to legal tender laws, the note is a debt instrument, which means it may be discharged by tendering Federal Reserve Notes in place of whatever the debt actually calls for.
So you go to the Fed and ask to redeem a $100 bill, and they will hand it right back to you. Technically speaking, they should give you a different $100 bill, other than the one you are trying to redeem, but really, if you get that far, you're already lucky they aren't physically throwing you out of the building.
This is one of those "laws and sausages" things. You're far better off not looking at it too closely if you want to continue to enjoy the benefits unburdened by unnecessary knowledge.
If you don't believe that, try to pay your taxes with gold and see what happens.