But when communication doesn't fail, grammatical errors don't matter that much. Because communication success is its own evidence (cue your "but how do you know [...]?" -- please don't. We've been down that road and your argument is unconvincing and solipsistic).

You say grammar correctness is another uncontroversial measure of NLP, but nobody was arguing against this. Of course it is. You're shifting the goalpost. Grammar matters. You can tell when something is ungrammatical but whether this affects communication is another issue. It's also a matter of degree: something can be so ungrammatical it conveys no meaning; and something can be ungrammatical enough to sound "bad", but still convey its intended meaning.

> Hence, we're left with heuristics like "how close this automatic translation is to this human translation"

That's one heuristic, sure, and a pretty good one! Not the only one, though. In any case, you missed the point that your Math example was not very good, because it ignored that in a heuristic, what you proposed as a bad "translation" could actually be a good one. In the realm of human language translations, it's never a case of 4+4=8.

> As a P.S., I'm not "claiming" to be working on a PhD. The information is in my profile with links to my linkedin and github :)

My mistake then! No offense intended.

PS: one more thing: I've looked at your comments history and I actually agree with your skepticism about AI. I think it's overhyped, especially by people with no background and no formal understanding about it -- like Eliezer, who you rightfully mock in your profile, and many other Prophets of the AI Apocalypse (or AI Utopia, depending on their mindset). So consider me a skeptic as well. We're more in agreement that it seems! I just don't think the kind of translation we're talking about is anywhere close to General AI or "understanding" of any form, and I can get excited about a "limited breakthrough" even though it's not actually about AI :)