Remember this: everyone is taught their perfect grammar and spelling and quizzed on it for years in school. How many people can say what a past participle is, or even what an adverb is? How many people who passed geometry can still do that level of math from the top of their heads?
Of course you can't permanently learn something. You have to keep it in practice.
It's hard to master and complete a Duolingo course without getting a grip on the language. You'd have to pretty much ignore the "Strengthen your skills" button and never repeat anything.
Even after finishing a course you should use the repetitions for weeks and months to keep refreshing your memory. That's the only way it may stay forever.
I'd love to see a independent study into its reliability/retention. But even if it is completely useless, it is still no more wasteful than any other mobile game people play daily.
I have seen the effects myself. After a month or so, even though I had learned some spanish before, native speakers suddenly slowed down (in my perception).
An important clue: Use the "Strengthen your skills" button frequently to keep the tree golden. Duolingo itself doesn't nag you enough to do that, but it's essential for mastery of the language.
I spent a lot of time barely using certain foreign languages (+ than 2 years). Of course you get 'rusty', but far from forgetting everything.
And it came up back quickly
This tell me Duolingo doesn't make you learn it, but rather memorize their game.