I found the English as an intermediary language amusing when translating a word between two Slavic languages, Polish and Russian. Wanted to look up something about Russian hand planes (the wood working tool). Typed "strug" and "hebel" into Google translate, and got... "самолет", which, being able to parse some Cyrillic, reads suspiciously like Polish "samolot" (airplane). Same the other way around: "рубанки" (hand planes) -> "samoloty" (airplanes). It kept the plural form, but of the wrong word :) The "hand" part of "hand plane" is getting lost in translation, both ways.
Edit: similar between Polish and German, "hebel" -> "Flugzeug", all the more amusing considering hebel is a loanword from German Hobel :)
When I need to translate a single word I go to its Wikipedia page in language A, then look for language B in the "in other languages" list. (That's how I finally came to "рубанки").
Good point, but it's even worse than meaningless. It's stereotype-reinforcing. That particular inference, that the role in society is such-and-such, so presume that the subject is male, is to be fought at every turn.
I would much prefer the awkwardness of "he/she" when the machine is uncertain rather than some "this is just the way things are" presumption.
Which mean that can only ever be an agent of reinforcing existing "cultures" and never really evolve into an agent of exploratory learning.
Now it might not be a big problem in the real world that google's algorithms essentially exists as Kuhn paradigm enforcers, locking people into a bubble of accepted dogmatic knowledge(as traditional journalism plays/played the same role is), if it were not for the fact that just about everyone pretends the opposite is true for modern machine learning.
This is a bug. Plain and simple.
(Also as someone who speaks French I would like to point out that the counter on "English speaking person programs a language related thing that gets gender wrong" has long since crossed the 6 digit mark. Maybe language apps should be written by people whose native tongue is a complex language like French or German (I am sure there are others, just examples))
So unfortunately this is a fundamental bug that will take serious research to solve, and it's unlikely to be solved really quickly (let's say ... a year, two tops), but it's a bug that will be solved.
It's like "energy is too expensive to just have cheap global travel without airports" type bug. It's a bug, it's currently beyond our capability to fix, but it WILL be fixed.
The other claims you make are too far fetched to even respond to. All of Google's algorithms, bubble of accepted dogmatic knowledge (I would even contend that Google by itself is responsible for 20-30% of the worldwide IQ rise seen during it's existence).
> just about everyone pretends the opposite is true for modern machine learning.
Nope. That's not pretending. That's simply true.
However when you request such a machine algorithm "do the same those guys do", and "those guys" are a racist/sexist/... society, then it will come up with slightly off-color conclusions on occasion.
Humans, in my experience, are much worse.
But, I don't think he/she is viable because a major use case is people pasting quickly on their phones. You can see this because the app has automatic clipboard copy built in.
He/she has always been pretty awkward and silly. But in the eyes of at least some people, it also has the failing that even after all that it still assumes binary genders.
Curious, I looked for guidelines and the best I could come up with was this, which seems helpful if you know the person in question, but not so much if you're referring to someone completely unknown: https://en.wikipedia.org/wiki/Third-person_pronoun#Transgend...
I welcome advice.
Singular "they" has a long history of being used as a generic pronoun, before a bunch of white men decided in the 18th century to try and prescribe "he." Singular "they" is making a comeback, though, and in this case it would be appropriate.
Isn't it simply _necessary_ to do such things? Don't we all have to assume the common case whenever someone leaves some detail out, which is of course done all the time? If someone says "they are kicking the ball between each other", I'm going to assume that the ball is round, as is typical for the occasion. That is "stereotype-reinforcing", but could we even avoid it? Would we want to?
Maybe the former expresses your intent better (for some reason?), but why not use the latter?
This, of course, is quite aside from what a machine should do in ambiguous cases. In those cases, I just wanted to put a vote in for things like he/she (or: he or she) coming from machines rather than statistics-based presumptions.
The point here is simply that Google's algorithm is translating texts and applies a statistical bias, i.e. the algorithm introduces systematic errors into translations. This results in objectively poor performance of a translation system.
https://www.quora.com/Does-Google-Translate-use-English-as-a...
It appears that they have been trying to construct (with ML, not by hand) a universal intermediary language since at least 2016 to avoid this sort of thing, but that it still isn't as good as just using English as the intermediary.
https://www.newscientist.com/article/2114748-google-translat...
And it's hard even for them to know because their models are opaque and hardly amenable to principled examination.
(I mean, you can't prove anything about them, so you can't really know what they're doing in there).
Oops. Google has enough money to hire some linguists, so they hopefully have heard of the debates over Chomsky's Universal Grammar. "Universal human language" is highly unlikely to work; it doesn't even work well for color names.
I have a couple of these in Japanese. One is "七輪" (brazier), which Google Translate thinks is "tambourine"; the other is "ちゃぶ台" (tea table), for which it returns "Shabu-bashi". No idea what's going on in either case - the latter doesn't even mean anything in Japanese.
It's always nice to read Hofstadter, but when Google Translate trips up over single words, delving into subtleties of contextual grammar seems like overkill.
Ah, but see, the very counterintuitive reasoning is that it's actually easier to translate a sentence than a single word- because the context provided by the other tokens in the sentence can help determine the intended usage of the word.
For instance, the following sentence is translated pretty much 100% correctly by GT:
Το χελιδόνι είναι αποδημητικό πτηνό.
The swallow is a migratory bird.
Of course, when sentences start getting more complex, looking at the near context, or even the far context (that LSTM nets are good at) is not going to help you. That's even more so when you have no way to tell which of multiple possible meanings of a word you should use to derive the correct context and find the correct translation.
So, yes, the inability of GT to discern between meaning at the word-level affects its ability to translate more complex passages, but you may miss that with simpler sentences- which I think may explain why many people think that GT translations are actually pretty good, when they mostly really suck.
In other words they're cases where GT is less useful than a plain lookup database - which makes comparisons to human translators seem rather beside the point!
However, it's often very hard to rev-eng the errors committed by statistical machine translation (and, generally, language processing) systems because they're basically just vectors of numbers that are completely meaningless to us. Which also makes it very hard to identify patterns of errors and correct them.
Another thing that's very wrong with machine translation is the measures of "goodness" of translations. This is usually something like the BLEU score, that work by comparing the words in machine translation to those in human translation. Those are very problematic in all sorts of ways, not least the inability to select a "good" candidate translation among the many possible.
In other words, it's not just that gathering statistics over words in context is not a very good way to model translations. On top of that, the results of such translations are very difficult to evaluate objectively.
In general, machine translation is a very vaguely defined problem- which lets GT claim to be really, really good at it even when it's easy to see it really, really isn't.
No, it doesn't. It goes via an intermediate representation which is common for all the languages they translate. The meaning of a phrase is represented as a "meaning vector", or more recently, a matrix. The problem is that it is hard to translate rare words, because when the neural net decodes the next word, it needs to consider the whole dictionary at once (the problem of too large softmax function slowing down the process). So they need to focus on a subset of words. There are various solutions, such as hierarchical softmax, but they are still difficult to train. Google translate uses a ton of compute to create a model as is. They probably can't offer everyone a version that is 2% better but uses 100x more resources.
Similarly, I would pay a lot of money for a Spotify or Netflix alternative where a knowledgeable human looked at my activity every month or so and created a hand-tailored recommendation list, since those products seem pretty bad at nuance in ways that even just simple "do you like this for reason A or reason B" questions would answer.
I know how the sausage is made at some non-Google places, and there's a common institutional blind-spot where the algorithms people don't want any human-driven chocolate getting in their peanut butter, and lazy product people are perfectly content to let any junk the algorithm throws up be the math guys' fault. Sadly, the world is full of edge cases, and sometimes you need one-off rules. (There's a parallel thread going below where people are aggressively defending the fact that the translate product as implemented is bad at certain use cases. Baffling. At the end of the day, are you happier telling users they're doing it wrong than given them what they want?)
Edit: I did not want to complain about the translation quality nor show my sense of entitlement. You have authoritatively stated and used a some jargon to tell us it doesn't use English when translating between languages, so I simply wanted to understand more about this process because, on the surface, it looks like English was involved in my specific example.
I can't say I can point to any sort of evidence of this, so take it with a pinch of salt, but my gut feeling is that the people in Google who are responsible for GT are somewhat fanatical about making purely statistical machine translation (and, in particular, deep-learning machine translation, a.k.a. neural machine translation) work. They are most probably perfectly well aware that it is possible to improve their system by using some rule-based fall-backs, or some background knowledge, but they have decided that they dont' want to use those and instead want to do machine translation from scratch and from data only, without any expert input.
So it sucks, like all pure systems suck.
This is like complaining that Word isn’t as good as vi, which is decades old, for text editing. Sure they could bolt a text editor onto Word if they wanted to, but why bother? That’s not what people are looking for when they use the product. And if that’s what you want, there are many other choices that are much better.
I would start by determining the number of words in the query. If it's equal to 1, the user probably wants to translate a single word, maybe give him the 1:1 dictionary result. If there's more, the other words possibly provide context, so let's use the sophisticated thing.
https://www.bloomberg.com/news/articles/2018-02-02/translati...
GT Engineers don't want a pat on the back. They want to do better.
Which makes a lot of sense when you realise that all the languages in the language pairs that GT's interlingua covers will have many, many more texts going to and from English, than to and from any other language. Attempting to derive a many-to-many language model from an English-dominated corpus is very likely to lead to an intermediary model that is basically English-to-many.
In fact, if you read the relevant paper [1] the dominance of English in tables etc jumps out at you.
And if you look at how Google figured out its Neural Machine Translation system has created an interlingua, that is ... visualisation. So it's not like they have some kind of principled way to demonstrate that this interlingua they advertise actually exists. They just look at some pretty images and interpret them to say that it does.
Even this visual "evidence" as they call it, is from a many-to-many language model trained on Japanese, Korean and -you guessed it- English.
__________________
It don’t want to say google translate is a complete garbage. On the contrary, I use it often e.g to plan my holiday trips or shop online in other EU countries. Nevertheless, I rarely translate anything to my native language, but rather to English.
It the auto translation made a great progress and it is very useful, but it has a long way to go still.
I have Mandarin speaking friends (graduates of US universities), for whom English is a second language. They often use "she" instead of "he" (or vice versa) when referring to others.
If humans, who have been using English for 7-8 years, can't get this right, then I wouldn't call it a "big mistake". It is a mistake, for sure; but it's not the end of the world.
For humans, anaphora resolution is child's play. For computers, it's very tricky.
Of course that's not what I mean above. Yes, the string I present is an example of the nonsense that comes out of purely statistical machine translation. It's a simple example and a very simple sentence, but that sort of noun-pronoun disagreement can completely distort the meaning of more complex sentences, not to mention longer passages where there are references to multiple persons across sentence boundaries.
Here's another example (original text by myself):
Πήγαμε με τη Βασιλική να δούμε το Μήτσο στο νοσοκομείο. Είχε σπάσει το πόδι του
και του βάλανε λάμες. Η Βασιλική, που είναι λίγο καψούρα μαζί του, του πήρε κάτι
σοκολάτες και λουλούδια, σα χαζό κοριτσάκι από καμμιά ρομαντική σειρά. Εγώ πήγα
για συμπαράσταση λαέ. Το χειρότερο ξέρεις ποιό είναι; Όχι μόνο ήτανε εκεί η μάνα
του, ο πατέρας του κι η αδερφή του, αλλά ήτανε κι η Μαρία, που τά 'χανε πέρσι.
Καταστροφή!
We went with Vassiliki to see Mitsos in the hospital. He had broken his leg
and he put blades on him. Basilica, who is a little bit of mascara with him,
got something chocolates and flowers, like a stupid little girl from any
romantic series. I went for support. Worst you know who it is? Not only was
the mother there his father and his sister, but she was Maria, who lost her
last year. Destruction!
Me and Vassiliki went to see Mitsos at the hospital. He had broken his leg
and he had to have metal plates inserted. Vassiliki, who has a bit of a crush
on him, got him some chocolates and flowers, like a silly little girl from some
rom-com. I just went along for solidarity. And you know what the worse part is?
His mother and father and his sister were there and not just them, but also Maria
his ex from last year. Disaster!
It's also easy to see that we don't use statistical models to solve maths problems, because we don't need them: we know the rules of arithmetic etc that lead to the correct solutions. With translation, we don't know what those rules are (if they can even be represented as rules) so we go for statistical models instead- but that is only because we can't do anything better. Statistical machine translations are fundamentally flawed. They might be "good enough" for some situations, but they are effectively designed to fail, by not even trying to solve the actual problem of translation.
And so what they produce is nonsense- and it is nonsense even when it looks like it sorta, kinda makes sense. It's nonsense because it's basically, just random.
That's huge, it's leaps and bounds beyond what I could even begin to achieve without months of training before. I have literally read and made basic sense of signs, posters and menus in greek without even being able to read the alphabet. How is this not substantial progress on "the actual problem of translation"?
He had broken his leg and he put blades on him.
Basilica, who is a little bit of mascara with him, got something chocolates and flowers.
Worst you know who it is?
she was Maria, who lost her last year.
Destruction!
This is not "good enough" (as opposed to perfect). That's just terribly bad. If you think it works for you that's only because you're so good at understanding language by dint of your being human (well- I assume).
Of course human translators get it badly wrong. I've read human translations that are not much better than that nonsense above, to be honest. But a reasonably competent human translator will never make the kind of elementary mistakes that automatic translation does. And those elementary mistakes will always escalate to the point that the meaning of longer passages will deteriorate to gibberish.
As to being able to understand basic signs, posters and menus in Greek- you can do that with a good travel guide also. And the travel guide will not claim to be a breakthrough Artificial Intelligence system, like Google claims for GT.
When machine translation makes mistakes on the other hand, the translation itself makes no sense. It's not just that its meaning is unconnected to the original; its meaning isn't.
If a system is 99% right, but the remaining 1% is catastrophically bad, it negates a lot of the part that is actually good. You might end up more confused by the machine translation than you were without it.
Machine translations is one of these systems where 99% good simply isn't good enough.
But the article had more subtle examples where important nuances are lost in translation, and those are impossible to pick up unless you speak the original language, but we don't speak the original language, because that was the whole point of using GT to start with, right?
Obviously, GT will get better at reducing the gibberish errors, but that just strengthens the trap, because now you will never think it's making any errors, because you lack the knowledge required to know what errors it makes!
I really, really disagree. If 99% lets me understand the gist of an idea, or instructions for something (of course: where there is no risk of harm), I'd take it -- even with the 1% of hilariously catastrophic mistranslations. This means I can go to Japan or Greece and just use Google Translate and be reasonably confident I can manage with a combination of this tool and my human-level understanding of language :) It's definitely good enough for me.
The very first example in the article was one where GT gets the words mostly right, but completely misses the point.
And if you miss the point, then the original text hasn't been translated. It's merely been decoded, this is the entire point of the article. Calling what GT does "translation" is wrong.
Yes, there's absolutely value in decoding signs and texts and simple phrases and directions. It allows you to go to a foreign country and get around without speaking the native language. Great. But a phrasebook can do the exact same thing, except it doesn't claim breakthrough AI machine-consciousness bla bla bla.
I'm not as good with words as Hofstadter, so here's his words from the end of the article that summarizes the problem quite nicely:
"I’ve recently seen bar graphs made by technophiles that claim to represent the “quality” of translations done by humans and by computers, and these graphs depict the latest translation engines as being within striking distance of human-level translation. To me, however, such quantification of the unquantifiable reeks of pseudoscience, or, if you prefer, of nerds trying to mathematize things whose intangible, subtle, artistic nature eludes them. To my mind, Google Translate’s output today ranges all the way from excellent to grotesque, but I can’t quantify my feelings about it. Think of my first example involving “his” and “her” items. The idealess program got nearly all the words right, but despite that slight success, it totally missed the point. How, in such a case, should one “quantify” the quality of the job? The use of scientific-looking bar graphs to represent translation quality is simply an abuse of the external trappings of science."
> [Google Translate] allows you to go to a foreign country and get around without speaking the native language. Great. But a phrasebook can do the exact same thing, except it doesn't claim breakthrough AI machine-consciousness
Wait, who is talking about consciousness? That's a different deal, and let me assure you I'm as skeptical as you appear to be. AI is a buzzword here, don't let it distract you from the effectiveness of the actual techniques.
Also, it stands to reason portable automated translation like your phone can do is not something a phrasebook can do. I can point to a random sign, take a picture of it and have Google Translate give me a helpful if quirky translation. I definitely cannot pull out a phrasebook and hope something matches what I'm seeing in the sign; it will take ages. So it's at least more convenient.
In fact, I've done this with Japanese instructions to assemble a scale model. Granted, here the problem domain was very constrained, and I already sort of guessed what I was trying to do -- and yet, Google Translate was orders of magnitude more helpful than flipping through a phrasebook. Would it be as helpful if I was in completely uncharted waters, where I can make fewer guesses? Probably not. Still, very impressive.
The particular paragraph is mangled to the point of being gibberish, yes, and noone with the option to choose between your translation or Google's would give Google's even a second glance. But if I don't, it's that, or a paragraph of text that I can just about identify as being Greek, but which is otherwise, well, Greek to me. But as you concede, with context and human understanding of language, I was able to pick out a few crucial chunks of meaning.
Whether it's good enough depends on what I'm trying to achieve. If I'm judging your skill as a writer and storyteller, then obviously not. But if we hung out with Vasiliki, Mitsos and Maria last year and you put that on Facebook, I'd at least know enough to send my sympathy to Mitsos and have a good chuckle over Vasiliki's romantic endeavours, which would probably be what you intended - - which is pretty much the definition of "good enough".
As for travel guides and Google Translate, as someone who's travelled extensively both pre and post Google Translate, there is nothing in the difference of experience that doesn't qualify as a breakthrough. I have had fully meaningful (if not exactly deeply philosophical) conversations with people entirely mediated by Google Translate. I'm sorry your experiences have been so different from mine.
You just discovered the poor quality of LSTM generated text. It might come as a surprise to you, but it's legend between AI researchers :-(
If it's not translation (such as in chatbots, image captioning, or computer generated text in general), it's even worse. Unfortunately we can't generate text that makes sense. We can only do it in very specific cases and with lots of errors. Other language models are even worse than LSTMs.
The problem is that automated translators don't have real world experience and can't think causally. Humans can, because we are in possession of a body and live inside a complex world. We can experiment to test our hypothesis, GT only sees text and can't test anything.
I actually disagree it's gibberish. I could understand enough to make sense of the situation (the part about blades in his leg puzzled me, but I understood about Maria being an ex girlfriend even if the sentence didn't resemble actual human language). This translation is by definition not gibberish.
> But if we hung out with Vasiliki, Mitsos and Maria last year and you put that on Facebook, I'd at least know enough to send my sympathy to Mitsos and have a good chuckle over Vasiliki's romantic endeavours, which would probably be what you intended - - which is pretty much the definition of "good enough".
Exactly!
You have a definition of gibberish? ~.^
If I can extract a reasonably accurate understanding from a paragraph, then by definition it's not meaningless nor unintelligible.
What I mean by that is that all the language structures we choose to convey a certain meaning are arbitrary and, by themselves, completely meaningless. We assign meaning to them by convention (and the fact that this convention keeps changing is why we have different languages and why even the "same" language sounds very different a few years down the line). It is when this convention is breached that our ability to understand the intended meaning is compromised.
I'd liken this to adding noise to a signal. The more a passage of text deviates from the conventional structures for the intended meaning, the less the information that can be safely extracted from it. The more the noise, the less likely that 100% of people hearing or reading the utterance will understand 100% of its indended meaning.
So in other words- it's still gibberirsh even if you can understand it. Because maybe you personally can understand it, but any number of other people will not- people with different language abilities, that is.
To draw an analogy- maybe you personally don't need a vaccine for chicken pox because you are immune to it. That doesn't make the vaccine unnecessary. Others don't have your immunity. etc.
No, I disagree with this premise. It is contrary to the definition of gibberish I'm using (and which I linked to). Since we disagree on this fundamental issue, the rest of this conversation is very difficult.
If all language is gibberish, like you say, then it's ok for Google Translate to produce gibberish. (Sorry, that doesn't make sense to me!)
So language utterances mean nothing by themselves. They only have the meaning we assign to them by convention. It is in this sense that I say that those utterances are gibberish.
This might sound weird to you, but if you think about it, most natural language processing tasks basically come down to performing a search for the meaning that is related to a given structure. What makes this search very difficult is that structures are inherently meaningless, so there's no logical reason why a given string might mean something particular. Essentially, we try to teach the NLP system to learn our conventions. But that is very hard to do, because we don't exactly understand how these conventions work, ourselves.
So it's not OK for GT to produce random gibberish- because there is only one set of strings of gibberish that is associated with the intended meaning of a sentence in an original language.
And, I guess that's my point. If GT's search fails to find the right string then it's not doing very well.
> So it's not OK for GT to produce random gibberish
Thankfully, it doesn't in most cases. If it were random, we wouldn't be able to understand it (please, don't ask anything that starts with "how do you know...?". I know. Let's drop that angle, please!) The chance of all of us understanding roughly the same idea from random gibberish is effectively zero. Hence, it's not random gibberish.
So this is something to wonder about: why is it that we all speak in a certain way? If we can understand language with unfamiliar structure, why does language follow set patterns? Why don't we all speak any old way we like?
Mind you, I'm not saying I have any answer to this. Because it's actually a major question. You could, for example, observe that people will often break from accepted conventions when speaking or writing and it's hard to even measure the deviation from accepted conventions "in the wild" (as opposed to more formal text or speech). And yet there does seem to be a set of patterns of utterances that we observe when speaking or writing in our respective languages. There are rules, in language, and these rules are observed. Why is that?
Understanding the answer to this question -what makes language, language, rather than random noise- could tremendously boost NLP tasks of all sorts. To be honest, I don't see anyone asking it though. Instead people have become comfortable with the paradox of the random noise (sorry, I insist) generated by computer programs, and treat it as an engineering problem to be solved, rather than an opportunity to understand human language by looking at its difference to machine language (if I may).
>> (please, don't ask anything that starts with "how do you know...?". I know. Let's drop that angle, please!)
I understand if you don't want to spend time discussing this, but I hope you understand that, unless you justify your thinking, you can't expect anyone else to agree with it.
In any case, "how do you know" is the fundamental question you have to ask if you want to advance your knowledge. That's how it all begins- by challengine assumptions (yours or others').
Not everyone will understand your hand-made translation, either. Human understanding works by consensus. Someone will be left out of this consensus almost always. It doesn't mean the automated translation is gibberish. If enough people understand the automated translation, then by definition it's not gibberish, because random noise doesn't carry enough information for you to get the idea about Maria, Vassiliki, the hospital or the broken leg.
> I understand if you don't want to spend time discussing this, but I hope you understand that, unless you justify your thinking, you can't expect anyone else to agree with it.
I have justified it. I asked you to drop it because we're running in circles, not because I don't have arguments. Your argument seems solipsistic to me: "but how do you know?" -- I know because it's my experience and because other people agree with me. How do you know your translation is good, anyway? Because it's what you think and because other people agreed with you. Because you managed to successfully -- as assessed by yourself -- communicate the idea you wanted.
To go back to an earlier example of learning to solve mathematical problems with machine learning- let's call it machine arithmetic. Say we developed a machine arithmetic system that consistently estimated the sum of 4 + 4 to be 7. That's consistently close to the correct answer, but it's also consistently not the correct answer.
So, to pose yet another question: a system trained to solve maths problems who got it almost right 100% of the time, would be considered pretty rubbish. Why is GT not? It's computing language, not maths, but its computation is still very often completely wrong in specific, measurable ways that really leave no room for subjective interpretation.
What matters for the purpose of human communication is the latter, not the former. Success at communication trumps your grammatical errors (where beauty is not concerned, of course. If you're going to argue automated translation produces ugly results, then you'll hear no argument from me!).
> Not trying to be offensive, but your definition of gibberish is subjective
You're not offensive; you're arbitrary. I provided you with a dictionary definition of gibberish, while your definition seems to be suspiciously ad hoc. You've essentially defined gibberish as "whatever output automated translation currently produces", which makes this debate pointless.
Math and natural language are different. Again, the standards are different, just as in your example with child's language. No, we do not judge a computer's ability to "translate" the same as we judge a young kid's ability to speak. There's nothing surprising about these two standards being different.
Your example with Math is not particularly relevant. However, since your background seems not to be computer science [1], maybe you're unfamiliar with heuristics: in the context of computer science, a heuristic is a technique where you take an optimization problem (e.g. "finding the minimum of something", "the shortest route to somewhere", etc) which is computationally untractable and provide a "good enough" answer which can be computed. The definition of what is "good enough" is, of course, relevant to the application of a particular heuristic in a specific context. In this sense, for some heuristic and its application, maybe 7 is a "good enough" answer even though the right answer is 8.
[1] This was written before I read in another reply of yours that you claim to be working on a PhD in Machine Learning. Fair enough! But even if you're familiar with heuristics, I think my point stands.
But grammatical errors cause communication to fail. Because they introduce (measurable) noise to the signal, which obviously makes it harder to extract information from it.
Btw, grammaticality is another uncontroversial measure of the performance of NLP systems that are required to generate language (like machine translation does). There is some theory behind the idea that well-formed sentences are inherently more intelligible, that will take a lot more to overturn than "it works for me".
To draw yet another analogy- homeopathy "works" for some people: they feel better when they take its concoctions. That doesn't make homeopathy objectively useful.
>> in the context of computer science, a heuristic is a technique where you take an optimization problem (e.g. "finding the minimum of something", "the shortest route to somewhere", etc) which is computationally untractable and provide a "good enough" answer which can be computed.
Yes, looking at machine translation as a heuristic is a good point. Like I say in other comments, the problem of machine translation has no good definition. That's because it's very difficult to define what is a "good translation" in a formal and principled manner. Hence, we're left with heuristics like "how close this automatic translation is to this human translation" (where the human translation is whatever happens to be available, rather than something chosen for its quality or other attributes).
This is actually exactly what machine translation systems do. They are made to optimise a measure of error over examples of human translation. So GT is actually designed to do what you say is not important, produce grammatically and syntactically correct language, like its examples are.
In that, it fails. It may achieve your goal of "communication" but that is not really a heuristic, it's more of a quality that you, personally (and obviously other people too) assign some value to. But it's far from a universally accepted measure of machine translation quality. You might say that "communication" is a business goal whereas matching human translations is an engineering goal.
As a P.S., I'm not "claiming" to be working on a PhD. The information is in my profile with links to my linkedin and github :)
But when communication doesn't fail, grammatical errors don't matter that much. Because communication success is its own evidence (cue your "but how do you know [...]?" -- please don't. We've been down that road and your argument is unconvincing and solipsistic).
You say grammar correctness is another uncontroversial measure of NLP, but nobody was arguing against this. Of course it is. You're shifting the goalpost. Grammar matters. You can tell when something is ungrammatical but whether this affects communication is another issue. It's also a matter of degree: something can be so ungrammatical it conveys no meaning; and something can be ungrammatical enough to sound "bad", but still convey its intended meaning.
> Hence, we're left with heuristics like "how close this automatic translation is to this human translation"
That's one heuristic, sure, and a pretty good one! Not the only one, though. In any case, you missed the point that your Math example was not very good, because it ignored that in a heuristic, what you proposed as a bad "translation" could actually be a good one. In the realm of human language translations, it's never a case of 4+4=8.
> As a P.S., I'm not "claiming" to be working on a PhD. The information is in my profile with links to my linkedin and github :)
My mistake then! No offense intended.
PS: one more thing: I've looked at your comments history and I actually agree with your skepticism about AI. I think it's overhyped, especially by people with no background and no formal understanding about it -- like Eliezer, who you rightfully mock in your profile, and many other Prophets of the AI Apocalypse (or AI Utopia, depending on their mindset). So consider me a skeptic as well. We're more in agreement that it seems! I just don't think the kind of translation we're talking about is anywhere close to General AI or "understanding" of any form, and I can get excited about a "limited breakthrough" even though it's not actually about AI :)
>> No offense intended.
... and no offense taken, absolutely.
We'll probably get the chance to argue again in the future, anyway, given that you're interested in the same subjects as I am :)
Gibberish means nonsense. If most people understand you, it's not nonsense.
Any conversation can easily become "philosophical" when Google Translate messes up and inserts random garbage instead of intended meaning, and people, instead of dismissing translation as garbage, decide to seek some deeper meaning in it :)
I think you're understating the quality of the automated translation here. It's by definition not nonsense if I can derive a pretty accurate picture of the situation by reading it. While flawed, it's not random gibberish. If it were random, I -- not knowing how to read Greek -- wouldn't be able to understand about someone being in the hospital because of a broken leg, about some friends visiting him, and about the social faux pas of the two girls being in the same room as the patient. Yes, some of the mistranslations are funny, but they are not nonsense.
If an automated translation is good enough for a human-level understanding of language to parse and derive a decently accurate picture out of it, while not understanding a single word of the original text, then that's an impressive achievement!
By the way, those of us who remember the initial results of early Google Translate should be impressed. It was truly, really unusable. It has gone a really long way, to the point I consider using it in places where I really cannot understand a single word or read the script... like in Japan, for example.
Is a machine that produces gibberish that's not too mangled to understand better than having no translation at all? It certainly is. However, Google's hype machine will not stop there- they consistently overstate the ability of their systems, by a very large margin.
If Google was saying "look, we have a machine translation system, it's a bit shit but it gets the job done" that'd be fine. But they don't say that. They pretend their system actually solves machine translation, for real, like AlphaZero solved Go. You just have to watch the triumphant press releases that come out of Google.
For instance:
https://research.googleblog.com/2016/11/zero-shot-translatio...
The post you linked to makes no such claim. The strongest claim to, well, anything, is that it produces "reasonable" translations between language pairs it has never seen.
This inspired us to ask the following question: Can we translate between a language pair which the system has never seen before? An example of this would be translations between Korean and Japanese where Korean⇄Japanese examples were not shown to the system. Impressively, the answer is yes — it can generate reasonable Korean⇄Japanese translations, even though it has never been taught to do so. We call this “zero-shot” translation, shown by the yellow dotted lines in the animation. To the best of our knowledge, this is the first time this type of transfer learning has worked in Machine Translation.
Because what they claim to do is really out there- it is something that not even humans can do. If they could actually do it then they would have solved machine translation for real.
And they haven't - like I say above, they seem to have misinterpreted a model of English-to-many (other languages) as an interlingua. And it's not even a very good model of English-to-many.
So the difference between hype and reality is astronomic and I feel very well justified to say what I say.
It's not gibberish. It's just very bad quality. But it helps me extract the meaning, which is not that far from what I do with some of the emails I receive in my day job -- trust me!
To take this a step further- if your child was speaking in such a manner you would be very, very concerned. But with GT it's OK? Why?
Google Translate is supremely useful in the absence of a better alternative. Nobody is contesting that human translation is better than Google Translate, but a human translator isn't always available. Google Translate is better than nothing at all.
A child's first many, many attempts at language are absolute gibberish, then slowly individual words, then combinations of words, then gradually more complex sentences etc -- gradually over several years. When you child first says "mama", the common response is excitement, not berating the child and anyone excited that "mama" isn't Wordsworth, and asking if just saying "mama" is acceptable, why aren't we all just communicating using single words all the time?
A child being able to express itself at all using language is better than it not being able to, and cause for celebration, and so too it is with Google Translate.
I think we're talking at cross-purposes here a bit. I don't really care whether GT is useful as a tool. I care about whether the translations it offers are good translations or not. I think you're basically saying that, if it's good enough for you, then it's a good translation. However, this is not a good measure of the quality of a translation! To begin with, what's good for you is a sujective measure. Further, we wouldn't accept "it's good enough for me" for any other translation than a machine translation- and it's accepted for GT only because we understand the limitations of machine translation systems.
So basically my pointing out the limitations of machine translation systems has nothing to do with whether it's useful to you. It's an attempt to explain why the results of machine translation are objectively bad.
I'll agree if you want that it's very difficult to get to such an objective measure of goodness or badness. But that's actually a reason why it's so hard to do machine translation properly: because it's next to impossible to evaluate machine translation objectively, which in turn makes for a very poorly defined task.
So you can feel justified to say that GT is OK for you, because you can understand it. Others can say that it sucks because they can't. And Google can say it's a breakthrough in AI because it scores high on their formal tests, evaluating using BLEU score (or something equally arbitrary).
But none of those tells us how good or bad GT translation really is. So we're left with what Hofstadter does (and what I did here): eyballing it. And, eyballing it, you can see that it's making elementary, egregious mistakes that are not justified by the hype surrounding GT as a product.
Btw, we can expect a child to get better at using language because most children do. So far, no machine translation system has got a lot better than GT- and I don't even agree that GT has gotten any better over the years, or that it's even any better than earlier systems. It just happens to be Google's system and they advertise it as the best ever- but they can't actually, you know, prove this.
Because if, past a certain age, my child keeps talking this way, it's a sign of development problems. Google Translate is an algorithm and I don't worry about its mental health. The standards for software/machines and human children are different and I find nothing surprising about this fact.
I don't think anyone is claiming Google Translate "is OK" in the sense of "we're done here, we've done all we could, let's stop thinking about automated translation and declare it mission accomplished".
Two other comments accuse me of letting perfect be the enemy of good. But for this to happen, GT translations must be "good" in the first place. Well, they're not. They are often terrible.
I don't understand why this criticism is even controversial. Terrible won't become good if we don't point out that it is, in fact, terrible.
Plus, I'm very concerned about promoting commercial products as examples of technological and scientific progress. Of course Google has an incentive to claim its translation service is great. Why does anyone else?
Because the standards for automated translation, something that didn't exist a few decades ago, are different from the standards of human speech and translation? No-one is saying Google Translate is on par or anywhere near human translation.
> Terrible won't become good if we don't point out that it is, in fact, terrible.
Yes, but you see complacency where there is none. The engineers who achieved this are rightfully proud about it, but I doubt they think this is the end of the line for automated translation.
It's helpful to say "this is flawed here and there, you can do better!", but your attitude seems needlessly antagonistic. Frankly, it sounds to me as if you feel threatened by this development. You shouldn't. As you point out, this is miles behind actual human translation in subtlety. Human-level understanding of speech (with double meanings, play on words, puns, understatements, etc) may very well be impossible for software to achieve; in fact, I think it's forever out of reach. But that's not the goal of automated translation; it's merely meant to be a useful tool when there are no human translators available.
> Plus, I'm very concerned about promoting commercial products as examples of technological and scientific progress. Of course Google has an incentive to claim its translation service is great. Why does anyone else?
Because they are definitely examples of technological progress and we can feel excited about it, and hope it gets even better! I couldn't care less about the Google brand. I understand your concern and I'm not excited about Google-the-business.
Well, they kind of do. Like I say elsewhere, the common measure of machine translation performance is a family of metrics like BLEU or ROUGE that basically compare machine translation to human translation. So when a research team claims strong performance in machine translation, they're really saying that their systems can do the kind of translation that humans can do, in a very literal sense (of similarity between tokens or n-grams). Besides, like Hofstadter says, Google has often claimed its NLP systems (for example, its NL Parser, Parser Mc Parseface) approach or outdo human performance.
Google is quite unrealistic in the way it promotes its technology and claiming it does better than humans is its bread and butter in this regard.
>> Frankly, it sounds to me as if you feel threatened by this development.
I don't have any reason to feel threatened :) I'm a machine learning PhD researcher with some hands-on experience of NLP (though not machine translation specifically- I've only theoretical knowledge of it). I have a background in foreign languages and translation, but not professionally.
If I can summarise my concerns, it's not that I'm worried that machine translation will become so good it will take the jobs of human translators. I'm worried that we will keep replacing human workers with bad AI and end up making our lives a lot worse in the process.
"Я и Вассилики пошли проведать Митсоса в больнице. Он сломал ногу и ему поставили металлические вставки. Вассилики, которая немного влюблена в него, принесла ему шоколада и цветов как глупая девочка из романтического сериала. Я пошла только за компанию. И знаете что было хуже всего? Не только его мать и отец были там, но ещё и Мария - его бывшая с прошлого года. Ужас!"
"I and the Vassiliki went to see Mitsos in the hospital. He broke his leg and put metal inserts. Vassiliki, who is a bit in love with him, brought him chocolate and flowers as a stupid girl from a romantic TV series. I went only for the company. And you know what was the worst? Not only his mother and father were there, but also Maria - his ex from last year. Horror!"
The lack of understanding in statistical machine translation is a counter-argument I use whenever someone claims that its apparent success is a sign that the AI singularity is imminent, but I have to admit that the recent progress in statistical AI has led me to wonder if, just maybe, these methods are closing in on the fundamental methods of understanding. Perhaps John von Neumann's famous quip "Young man, in mathematics you don't understand things. You just get used to them" is actually generally true.
I am honestly surprised that a company with such talent creates a tool that is this useless; it's literally about doing something like this:
1) Check if the word or phrase is in a dictionary. If it is, return all results from the dictionary. 2) If it's not, try something smarter.
But no, this is not Artificial Intelligence enough! /s
...better to see Schank over Cherniak
The word is "χελιδόνι", meaning "swallow" (the bird) in Greek. For as long as I've been trying this tiny little experiment, Google has been translating it to the French word "avaler"- the verb "to swallow".
https://translate.google.com/#el/fr/%CF%87%CE%B5%CE%BB%CE%B9...
Once in a while, a different translation appears- "machaon", which is a kind of butterfly, the Old World swallow-tail. This is slightly closer but still absurd. If GT can make the connection to "swallow-tail", how can it not see the connection to the bird, "swallow"?
The problem seems to be that, in order to translate from Greek to French, Google goes via English. A great big chunk of context is lost in the process, especially since now the translation between two languages that have different forms for male and female nouns goes through a third language that does not.
So for example, I've seen the same gender-reversal as Hofstadter reports in his article. The following string in Greek means "I saw my teacher and she said hello to me".
The following are the French and English translations by GT: This seems to happen because Google's English language model has learned that "he" is found in the context of "teacher" and "said" more often than "she" is. But of course, such a statistical association is, well, meaningless and for that useless when it comes to translation, where you actually need to know when the least likely case is correct.Generally, it looks like the complete abandonment of any attempt at representing meaning, and relying instead on text statistics to do meaning-intensive work like translation, is producing a lot of nonsense. And that's not a criticism of Google Translate only. I think professor Chomsky might "win" that old debate with prof. Norvig, after all.