If I use e.g. Bernstein's cryptosystem and he's evil, then he and whoever hires him can read my data.
If I use my own or your cryptosystem, then either the cryptosystem or implementation definitely (with a much, much greater confidence than any trust issues) is horribly broken due to some bug, oversight or side channel, and I just haven't noticed yet. So the end result is that everyone can read my data - sticking with someone evil would have been more secure than rolling my own.
In that light, it'd be perhaps better to phrase trusting AES as "not distrusting the union of everyone who's tried to cryptanalyse AES", rather than trusting the Rijndael team specifically. In many cases, especially djb-brand crypto, there's even less need for trust: the way you derive the X25519 curve is extremely well defined (or, as djb puts it, "rigid").
The 'don't roll your own crypto' argument is mostly just shorthand to 'defer to the opinion of experts, use ready-made constructs when possible, and if not, then exercise caution when hooking crypto primitives together in unproven ways'. [1]
Crypto code, like other library code, is question of trust.
Do I trust Daniel Bernstein? Do I trust Joan Daemen, who is half of the AES team and a quarter of the Keccak team [2]? In practical matters, do I trust tptacek [3]? Yes, I trust them, until people more educated in cryptography than me cast enough doubt or prove it otherwise -- but you might have a different model of who you trust. But ultimately, you're the one who answers to your systems.
It's also a bit like science where we come up with a hypothesis (this seems to work...) and then try very hard disprove it, so our knowledge evolves as we go. It's important to understand that there was a time when it was best practice to use certain primitives that are now considered broken, and this is okay -- assuming we all upgraded our systems since. For cases then that assumption can't hold true, we need to account for that risk.
This is also why it's wise to go with cryptosystems that receive a good amount of peer scrutiny. Your homegrown secret sauce might indeed be super secure, but few will publish papers on how they can obtain collisions on a round-reduced version. It's network effects, it's 'given enough eyes' all over again.
[1] https://news.ycombinator.com/item?id=12400040 [2] https://news.ycombinator.com/item?id=12766941 [3] https://gist.github.com/tqbf/be58d2d39690c3b366ad