The consensus seems to be that you're doing encrypt-and-mac, where the mac is just a sha1? If that's true, then you're relying on broken properties of IGE (not provably UF-CMA), and your protocol does not provide provable integrity. That's an essential part of any secure protocol.
> As for KDF, going for slower provable algorithms used for each incoming\outgoing packet may be a preferred solution for projects aimed at the relatively small security crowd. But we don't really compete in this area, our competition is WhatsApp and other mass market messengers.
That's interesting, because the thing you've made up is actually slower than a provably secure KDF.
Now, again, could I somehow direct your attention away from speculations and to what we are ACTUALLY doing (as, again, documented here [0], and now here as well [1]).
1. We are not doing plain encrypt-and-mac.
2. The SHA1 in question is for raw unencrypted data.
3. The message key is SHA1-dependent.
4. Note that the AES key and iv depends on that SHA1.
This can be described as a generic composition of cipher with ciphertext, encrypted by a MAC. The resulting data-dependant variable key denies all common attacks.
As for KDF, what particular solution do you have in mind? And even then — certainly, alternative solutions exist, but we do not see how changing this point would affect our system as whole. [1] As stated before, we'd welcome any information on attacks that could in reality threaten the actual setup.
[0] - http://core.telegram.org/mtproto/description [1] - https://core.telegram.org/img/mtproto_encryption.png
All popular KDFs (PBKDF2, bcrypt, scrypt) are adaptive (tunable). You choose the number of iterations.
What's this mean? You've created your own KDF, and the questions were: why is it secure, and why didn't you use an already-proven KDF? Unless I'm misreading something, it sounds like you've responded, "It's not that important, because our target market isn't people who care about their security." Is that correct?
EDIT: Judging by the upvotes, at least 5 other people are interested to hear your thoughts on this. Considering this submission is almost off the frontpage, that's a lot. As a security product, you should consider clarifying your position on this, because your statement currently sounds like, "We don't think security is a big deal since it's not a big deal to our target market."
Even as a complete newbie in cryptography I can see a problem in your reasoning. "Guilty until proven innocent" is the default when judging on a system. "No attack was named" because Telegram uses algorithms combination that was not proven with time. Nobody used them like this before. Why would anyone try to break it if it wasn't used?
As a potential user of Telegram I don't trust it enough to use it. It's more likely that someone with resources (e.g. government) could eventually break your system than it could break another - time-proven - system.
As for KDF, going for slower provable algorithms used for each incoming\outgoing packet may be a preferred solution for projects aimed at the relatively small security crowd. But we don't really compete in this area, our competition is WhatsApp and other mass market messengers.