TextSecure is also using a protocol of its own design. I think the difference is largely the motivation at the other end. Moxie is genuinely engaged and interested in crypto and, much much more importantly, improving the trust models associated with it (see Convergence as another example). The Telegram guys seem more interested in being a 'hip' app with the latest secure IM solution. This doesn't even necessarily mean someone with far far less crypto knowledge than Moxie can't make a secure solution, it just means Telegram are suffering from a little arrogance and over-eagerness.

It's far too easy for people outside crypto circles to see cryptography as a panacea. Inside crypto circles however, it's my impression that everyone lives with a slight unease that much of the math they rely on has unproven lower bound complexity, the majority of implementations in existence are horrific, and the key management and trust models we all depend on are terrible and obscured from the users view and understanding. If you pay attention, all the rock star cryptographers spend most of their time talking about trust, not algorithms and protocols.

Designing a simple and secure crypto protocol isn't actually that hard. There are mathematical pitfalls us mortals can't hope to understand, but if you trust and understand the primitives as black boxes, and have the right mindset when analysing protocols, you can still build very secure systems. I'm a casual crypto hobbiest, and still spotted the issues raised in the Telegram protocol as soon as I looked at the diagram presented in the article. None of these weaknesses are outside of a good programmers comprehension.

So why is a custom protocol a bad idea? The same reason it's a bad idea to go and reimplement any other protocol... that problem has been solved, why are you remaking it without a strong incentive?

There are examples of crypto being used by amateurs with success though. Bitcoin has multiple extensions, like deterministic hierarchical wallets, which are easy to understand and reason about but I know for a fact weren't designed by world class crypto-experts. In that case, there was a strong incentive. Asynchronous key splitting to ensure safe generation of vanity addresses by 3rd parties is another example. Nobody should say these solutions aren't innovative and useful.