You have to trust the people who figured out the math, when they say "this math is secure". Then you have to trust the people who programmed that math, when they say "this code implements this math correctly". And then you have to trust the binary, unless you compiled it yourself, when it says "this binary compiles this code correctly", and finally you have to trust the machines when they say "this computer executes this binary correctly", all the while trusting youself not to have made any mistakes at any step which will quickly cost you everything you put into the system.

If you are an expert mathematician, programmer, electrical engineer, and white hat you might not have to trust anyone else. How many people are all of these though?