For the past 5 years or so, as a kind of benchmark, I've been checking how Google translates a particular word from Greek, to French.
The word is "χελιδόνι", meaning "swallow" (the bird) in Greek. For as long as I've been trying this tiny little experiment, Google has been translating it to the French word "avaler"- the verb "to swallow".
Once in a while, a different translation appears- "machaon", which is a kind of butterfly, the Old World swallow-tail. This is slightly closer but still absurd. If GT can make the connection to "swallow-tail", how can it not see the connection to the bird, "swallow"?
The problem seems to be that, in order to translate from Greek to French, Google goes via English. A great big chunk of context is lost in the process, especially since now the translation between two languages that have different forms for male and female nouns goes through a third language that does not.
So for example, I've seen the same gender-reversal as Hofstadter reports in his article. The following string in Greek means "I saw my teacher and she said hello to me".
Είδα τη δασκάλα μου και μου είπε γειά.
The following are the French and English translations by GT:
J'ai vu mon professeur et il m'a dit bonjour.
I saw my teacher and he said hello.
This seems to happen because Google's English language model has learned that "he" is found in the context of "teacher" and "said" more often than "she" is. But of course, such a statistical association is, well, meaningless and for that useless when it comes to translation, where you actually need to know when the least likely case is correct.
Generally, it looks like the complete abandonment of any attempt at representing meaning, and relying instead on text statistics to do meaning-intensive work like translation, is producing a lot of nonsense. And that's not a criticism of Google Translate only. I think professor Chomsky might "win" that old debate with prof. Norvig, after all.
> to translate from Greek to French, Google goes via English.
I found the English as an intermediary language amusing when translating a word between two Slavic languages, Polish and Russian. Wanted to look up something about Russian hand planes (the wood working tool). Typed "strug" and "hebel" into Google translate, and got... "самолет", which, being able to parse some Cyrillic, reads suspiciously like Polish "samolot" (airplane). Same the other way around: "рубанки" (hand planes) -> "samoloty" (airplanes). It kept the plural form, but of the wrong word :) The "hand" part of "hand plane" is getting lost in translation, both ways.
Edit: similar between Polish and German, "hebel" -> "Flugzeug", all the more amusing considering hebel is a loanword from German Hobel :)
When I need to translate a single word I go to its Wikipedia page in language A, then look for language B in the "in other languages" list. (That's how I finally came to "рубанки").
"...such a statistical association is, well, meaningless..."
Good point, but it's even worse than meaningless. It's stereotype-reinforcing. That particular inference, that the role in society is such-and-such, so presume that the subject is male, is to be fought at every turn.
I would much prefer the awkwardness of "he/she" when the machine is uncertain rather than some "this is just the way things are" presumption.
And you just hit on one of the major problems with practically all of google's algorithms, in a social context, as pretty much everything Google ever does is measuring popularity(and for the most advanced of their algorithms popularity within "subculture"), and it's not like Facebook is really doing anything more sophisticated then ranking popularity and putting people in boxes either.
Which mean that can only ever be an agent of reinforcing existing "cultures" and never really evolve into an agent of exploratory learning.
Now it might not be a big problem in the real world that google's algorithms essentially exists as Kuhn paradigm enforcers, locking people into a bubble of accepted dogmatic knowledge(as traditional journalism plays/played the same role is), if it were not for the fact that just about everyone pretends the opposite is true for modern machine learning.
Isn't this a serious stretch ? I mean, sure there currently is an algorithm limitation that "turns it into a popularity contest". (I strongly disagree with that wording, but I sort of see what you mean if I squint strongly)
This is a bug. Plain and simple.
(Also as someone who speaks French I would like to point out that the counter on "English speaking person programs a language related thing that gets gender wrong" has long since crossed the 6 digit mark. Maybe language apps should be written by people whose native tongue is a complex language like French or German (I am sure there are others, just examples))
So unfortunately this is a fundamental bug that will take serious research to solve, and it's unlikely to be solved really quickly (let's say ... a year, two tops), but it's a bug that will be solved.
It's like "energy is too expensive to just have cheap global travel without airports" type bug. It's a bug, it's currently beyond our capability to fix, but it WILL be fixed.
The other claims you make are too far fetched to even respond to. All of Google's algorithms, bubble of accepted dogmatic knowledge (I would even contend that Google by itself is responsible for 20-30% of the worldwide IQ rise seen during it's existence).
> just about everyone pretends the opposite is true for modern machine learning.
Nope. That's not pretending. That's simply true.
However when you request such a machine algorithm "do the same those guys do", and "those guys" are a racist/sexist/... society, then it will come up with slightly off-color conclusions on occasion.
But be honest, language has this odd 'gender' thing going on. Orthogonal to any interests about equality of the sexes. Not saying its right or helpful, but its there and a responsible translation algorithm has to take it into account.
Agreed. I edited my comment to add what I think would be a reasonable way to handle it. "he/she" is just silly for humans to say, but is seems appropriate when the text is coming from a machine and there is uncertainty.
Makes it hard to paste though. Then again, so does an error in gender.
But, I don't think he/she is viable because a major use case is people pasting quickly on their phones. You can see this because the app has automatic clipboard copy built in.
Singular they seems to be the general approach that people are moving towards. It's now accepted in AP style under some circumstances--the some circumstances being basically if you can't figure out a way to cleanly write around it. And AP style is one of the stylebooks a lot of organizations look to for their own house styles.
He/she has always been pretty awkward and silly. But in the eyes of at least some people, it also has the failing that even after all that it still assumes binary genders.
In the context of this thread, I thought: if I don't want the machine to presume male or female, than what do I want it to do? he/she was what I came up with, but this does not address non-binary gender or fluid gender, weaknesses which were on my mind.
Curious, I looked for guidelines and the best I could come up with was this, which seems helpful if you know the person in question, but not so much if you're referring to someone completely unknown: https://en.wikipedia.org/wiki/Third-person_pronoun#Transgend...
Singular "they" has a long history of being used as a generic pronoun, before a bunch of white men decided in the 18th century to try and prescribe "he." Singular "they" is making a comeback, though, and in this case it would be appropriate.
Isn't it simply _necessary_ to do such things? Don't we all have to assume the common case whenever someone leaves some detail out, which is of course done all the time? If someone says "they are kicking the ball between each other", I'm going to assume that the ball is round, as is typical for the occasion. That is "stereotype-reinforcing", but could we even avoid it? Would we want to?
It certainly can be difficult to work around gendered language. The point is to think about it and avoid it when you can. You never met your daughter's surgeon: "I'm sure the surgeon did all he could to help her." "I'm sure the surgeon did all that could be done to help her."
Maybe the former expresses your intent better (for some reason?), but why not use the latter?
This, of course, is quite aside from what a machine should do in ambiguous cases. In those cases, I just wanted to put a vote in for things like he/she (or: he or she) coming from machines rather than statistics-based presumptions.
Could you please stop posting ideological flamebait to HN? Two of your last three comments have been that—this is bad. The other of the three was quite good, so if you stick to posting civilly and substantively, you'll be fine.
The point here is simply that Google's algorithm is translating texts and applies a statistical bias, i.e. the algorithm introduces systematic errors into translations. This results in objectively poor performance of a translation system.
It appears that they have been trying to construct (with ML, not by hand) a universal intermediary language since at least 2016 to avoid this sort of thing, but that it still isn't as good as just using English as the intermediary.
My understanding is that English is the intermediary language, but not on purpose- they tried to create an "interlingua" but they trained with too many English-to-X and X-to-English data and ended up with a model of English as their intermediary language model.
And it's hard even for them to know because their models are opaque and hardly amenable to principled examination.
(I mean, you can't prove anything about them, so you can't really know what they're doing in there).
> It appears that they have been trying to construct (with ML, not by hand) a universal intermediary language...
Oops. Google has enough money to hire some linguists, so they hopefully have heard of the debates over Chomsky's Universal Grammar. "Universal human language" is highly unlikely to work; it doesn't even work well for color names.
> ..I've been checking how Google translates a particular word from Greek, to French.
I have a couple of these in Japanese. One is "七輪" (brazier), which Google Translate thinks is "tambourine"; the other is "ちゃぶ台" (tea table), for which it returns "Shabu-bashi". No idea what's going on in either case - the latter doesn't even mean anything in Japanese.
It's always nice to read Hofstadter, but when Google Translate trips up over single words, delving into subtleties of contextual grammar seems like overkill.
>> It's always nice to read Hofstadter, but when Google Translate trips up over single words, delving into subtleties of contextual grammar seems like overkill.
Ah, but see, the very counterintuitive reasoning is that it's actually easier to translate a sentence than a single word- because the context provided by the other tokens in the sentence can help determine the intended usage of the word.
For instance, the following sentence is translated pretty much 100% correctly by GT:
Το χελιδόνι είναι αποδημητικό πτηνό.
The swallow is a migratory bird.
Which in a way makes the mistake on the single word even harder to understand (I mean, for users who don't know anything about how machine translation works).
Of course, when sentences start getting more complex, looking at the near context, or even the far context (that LSTM nets are good at) is not going to help you. That's even more so when you have no way to tell which of multiple possible meanings of a word you should use to derive the correct context and find the correct translation.
So, yes, the inability of GT to discern between meaning at the word-level affects its ability to translate more complex passages, but you may miss that with simpler sentences- which I think may explain why many people think that GT translations are actually pretty good, when they mostly really suck.
I understand what you're saying, but there's no subtlety or dual meaning in my examples - Google Translate is just taking in a word with a single, unambiguous meaning, and returning a wholly unrelated word (or in the latter case gibberish).
In other words they're cases where GT is less useful than a plain lookup database - which makes comparisons to human translators seem rather beside the point!
Fair point. My hunch is that there is in fact some subtle ambiguity associated with your example words, that is not immediately apparent for human spearkers of Japanese and English, simply because they have all sorts of "filters" that wring out the right meaning even for single words, filters that are missing from simple dumb statistical translation.
However, it's often very hard to rev-eng the errors committed by statistical machine translation (and, generally, language processing) systems because they're basically just vectors of numbers that are completely meaningless to us. Which also makes it very hard to identify patterns of errors and correct them.
Another thing that's very wrong with machine translation is the measures of "goodness" of translations. This is usually something like the BLEU score, that work by comparing the words in machine translation to those in human translation. Those are very problematic in all sorts of ways, not least the inability to select a "good" candidate translation among the many possible.
In other words, it's not just that gathering statistics over words in context is not a very good way to model translations. On top of that, the results of such translations are very difficult to evaluate objectively.
In general, machine translation is a very vaguely defined problem- which lets GT claim to be really, really good at it even when it's easy to see it really, really isn't.
No, it doesn't. It goes via an intermediate representation which is common for all the languages they translate. The meaning of a phrase is represented as a "meaning vector", or more recently, a matrix. The problem is that it is hard to translate rare words, because when the neural net decodes the next word, it needs to consider the whole dictionary at once (the problem of too large softmax function slowing down the process). So they need to focus on a subset of words. There are various solutions, such as hierarchical softmax, but they are still difficult to train. Google translate uses a ton of compute to create a model as is. They probably can't offer everyone a version that is 2% better but uses 100x more resources.
Lots of discussion about compute-intensive alternative solutions there, but what about a human layer where something triggers a "wow, this word is rare, let's call up Prof Whoever and ask them for a suggestion" notification? There's a lot of words in the world, sure, but you only need to solve each one so many times. Dictionaries seemed to manage this trick even before computers!
Similarly, I would pay a lot of money for a Spotify or Netflix alternative where a knowledgeable human looked at my activity every month or so and created a hand-tailored recommendation list, since those products seem pretty bad at nuance in ways that even just simple "do you like this for reason A or reason B" questions would answer.
I know how the sausage is made at some non-Google places, and there's a common institutional blind-spot where the algorithms people don't want any human-driven chocolate getting in their peanut butter, and lazy product people are perfectly content to let any junk the algorithm throws up be the math guys' fault. Sadly, the world is full of edge cases, and sometimes you need one-off rules. (There's a parallel thread going below where people are aggressively defending the fact that the translate product as implemented is bad at certain use cases. Baffling. At the end of the day, are you happier telling users they're doing it wrong than given them what they want?)
How do you get from Polish "hebel" to German "Flugzeug", if not by dropping the "hand" part of "hand plane"? Especially when the Polish is a loanword from German Hobel...
We've become too entitled and can't appreciate the huge amount of work that went into automatic translation any more. If GT doesn't cut it, we can always take a ... you know ... dictionary and do it by hand. Google search isn't perfect either, Wikipedia has errors, Twitter has bots, why are we bothering to use them? Because they are better than not having them.
Or maybe one could reasonably expect that Google, with all that money and engineering power, would implement something like a dictionary lookup to short-circuit single word translations?
Edit: I did not want to complain about the translation quality nor show my sense of entitlement. You have authoritatively stated and used a some jargon to tell us it doesn't use English when translating between languages, so I simply wanted to understand more about this process because, on the surface, it looks like English was involved in my specific example.
I don't say it's easy, at least from organizational/product-development perspective. But that's basically how every translation website before GT worked: type a word in language A, select language B, get the result(s) from a dictionary, done. In the nineties I had this pocket device: https://i.imgur.com/Yp9yTRL.png which would look up and translate words in 8 languages. Today I have a smartphone which can connect to a datacenter with enormous computing power, but a dictionary lookup is a hard problem?
I think the problem is cultural, not one of engineering.
I can't say I can point to any sort of evidence of this, so take it with a pinch of salt, but my gut feeling is that the people in Google who are responsible for GT are somewhat fanatical about making purely statistical machine translation (and, in particular, deep-learning machine translation, a.k.a. neural machine translation) work. They are most probably perfectly well aware that it is possible to improve their system by using some rule-based fall-backs, or some background knowledge, but they have decided that they dont' want to use those and instead want to do machine translation from scratch and from data only, without any expert input.
As a developer I kind of get it: "we have all this machinery so we won't compromise to use the simple route even if it makes sense short-term; let's improve our product instead". But as a user I'd expect Google to be at least as good as dictionary lookup from two decades ago, and do PL -> RU or PL -> DE instead of PL -> EN -> RU, PL -> EN -> DE when I type in a single word.
Google Translate simply isn’t meant to be used for dictionary lookup, though. If you use it for that then you’re using the wrong tool.
This is like complaining that Word isn’t as good as vi, which is decades old, for text editing. Sure they could bolt a text editor onto Word if they wanted to, but why bother? That’s not what people are looking for when they use the product. And if that’s what you want, there are many other choices that are much better.
or maybe single word lookups are pretty useless for most translations? How would you identify those words which can be directly looked up in a dictionary vs those that require the current context for translation?
> How would you identify those words which can be directly looked up in a dictionary vs those that require the current context
I would start by determining the number of words in the query. If it's equal to 1, the user probably wants to translate a single word, maybe give him the 1:1 dictionary result. If there's more, the other words possibly provide context, so let's use the sophisticated thing.
Dissatisfaction with the status quo is precisely what keeps automatic translation companies in business, and indeed drives all innovation. The author explicitly gives credit to GT for being good enough for lots of people most of the time, but that does not excuse its fundamental failures.
GT Engineers don't want a pat on the back. They want to do better.
Yes, I know about GT's "interlingua". I've chosen my "swallow" example specifically to show that it's basically just a model of English.
Which makes a lot of sense when you realise that all the languages in the language pairs that GT's interlingua covers will have many, many more texts going to and from English, than to and from any other language. Attempting to derive a many-to-many language model from an English-dominated corpus is very likely to lead to an intermediary model that is basically English-to-many.
In fact, if you read the relevant paper [1] the dominance of English in tables etc jumps out at you.
And if you look at how Google figured out its Neural Machine Translation system has created an interlingua, that is ... visualisation. So it's not like they have some kind of principled way to demonstrate that this interlingua they advertise actually exists. They just look at some pretty images and interpret them to say that it does.
Even this visual "evidence" as they call it, is from a many-to-many language model trained on Japanese, Korean and -you guessed it- English.
Are you sure that this particular translation is "nonsense"? To me, it seems almost perfect. The only mistake in the French version is the gender of the teacher, which is a mistake that many humans can and do make, in addition to many other mistakes...
Well, the Greek version says she-teacher and it translates to French he-teacher. Def a big mistake. Note this is not the same as messing up if eg some random butterfly you see is he or she in French. The context is there, but google translate seems to convert everything to English, loosing some meaning in the process.
It don’t want to say google translate is a complete garbage. On the contrary, I use it often e.g to plan my holiday trips or shop online in other EU countries. Nevertheless, I rarely translate anything to my native language, but rather to English.
It the auto translation made a great progress and it is very useful, but it has a long way to go still.
> Well, the Greek version says she-teacher and it translates to French he-teacher. Def a big mistake.
I have Mandarin speaking friends (graduates of US universities), for whom English is a second language. They often use "she" instead of "he" (or vice versa) when referring to others.
If humans, who have been using English for 7-8 years, can't get this right, then I wouldn't call it a "big mistake". It is a mistake, for sure; but it's not the end of the world.
Your friends however will be able to keep track of the references to the same person in a conversation, even if they get the gender wrong. In GT's case it's evidence that it can't do that, and therefore will mangle longer passages that require this level of understanding.
Actually, no. Just a couple of days ago, when talking about a female colleague, they kept switching from "he" to "she" and back and forth. It's almost as if Mandarin has no "he" or "she" equivalents.
I'm a french native (from Quebec) and to me, there is no mistake. We'd often say "Ma professeure" for the feminine. Maybe in France or other countries they are more formal. Either way, this isn't a good example of Google Translate limitation.
In a way, it's literally nonsense, in the, uh, sense that it can't makey any sense of what it's translating and is just choosing some output string at random (i.e. with probability drawn from some distribution it has derived from its training examples).
Of course that's not what I mean above. Yes, the string I present is an example of the nonsense that comes out of purely statistical machine translation. It's a simple example and a very simple sentence, but that sort of noun-pronoun disagreement can completely distort the meaning of more complex sentences, not to mention longer passages where there are references to multiple persons across sentence boundaries.
Here's another example (original text by myself):
Πήγαμε με τη Βασιλική να δούμε το Μήτσο στο νοσοκομείο. Είχε σπάσει το πόδι του
και του βάλανε λάμες. Η Βασιλική, που είναι λίγο καψούρα μαζί του, του πήρε κάτι
σοκολάτες και λουλούδια, σα χαζό κοριτσάκι από καμμιά ρομαντική σειρά. Εγώ πήγα
για συμπαράσταση λαέ. Το χειρότερο ξέρεις ποιό είναι; Όχι μόνο ήτανε εκεί η μάνα
του, ο πατέρας του κι η αδερφή του, αλλά ήτανε κι η Μαρία, που τά 'χανε πέρσι.
Καταστροφή!
Google translates:
We went with Vassiliki to see Mitsos in the hospital. He had broken his leg
and he put blades on him. Basilica, who is a little bit of mascara with him,
got something chocolates and flowers, like a stupid little girl from any
romantic series. I went for support. Worst you know who it is? Not only was
the mother there his father and his sister, but she was Maria, who lost her
last year. Destruction!
My intended meaning:
Me and Vassiliki went to see Mitsos at the hospital. He had broken his leg
and he had to have metal plates inserted. Vassiliki, who has a bit of a crush
on him, got him some chocolates and flowers, like a silly little girl from some
rom-com. I just went along for solidarity. And you know what the worse part is?
His mother and father and his sister were there and not just them, but also Maria
his ex from last year. Disaster!
Examples like this reinforce my intuition that, in translation, it's impossible to get correct results while starting from the wrong premises. If you think about maths, for example. I'm pretty sure that you can train a statistical model (a translation model, actually) to try and generate results of various mathematical expressions. It's easy to see that such a model would often get the wrong results, because of a lack of any understanding of mathematics. In fact, you could very accurately say that it would produce a lot of nonsense. And even if it got the right answer- what would that prove? It would still be obvious that it doesn't know how to solve maths problems, just guess at their more likely solution.
It's also easy to see that we don't use statistical models to solve maths problems, because we don't need them: we know the rules of arithmetic etc that lead to the correct solutions. With translation, we don't know what those rules are (if they can even be represented as rules) so we go for statistical models instead- but that is only because we can't do anything better. Statistical machine translations are fundamentally flawed. They might be "good enough" for some situations, but they are effectively designed to fail, by not even trying to solve the actual problem of translation.
And so what they produce is nonsense- and it is nonsense even when it looks like it sorta, kinda makes sense. It's nonsense because it's basically, just random.
This seems like an example of letting the perfect be the enemy of the good. Translation without any loss of veracity is extremely difficult, even for human translators, and failure to clear that bar doesn't constitute nonsense. If instead we consider that I don't know a single word of greek, nor even read the alphabet, I can put that text into Google translate, and with a minimum of contextual knowledge of Vassiliki and perhaps Maria, I can get perhaps 80% of the meaning of the paragraph with zero effort. Same deal for the sentence with the teacher; the number of situations where the gender of the teacher is the pivotal point of meaning in the sentence would be very low.
That's huge, it's leaps and bounds beyond what I could even begin to achieve without months of training before. I have literally read and made basic sense of signs, posters and menus in greek without even being able to read the alphabet. How is this not substantial progress on "the actual problem of translation"?
Let's focus a bit at some of the translationd of sentences in my passage:
He had broken his leg and he put blades on him.
Basilica, who is a little bit of mascara with him, got something chocolates and flowers.
Worst you know who it is?
she was Maria, who lost her last year.
Destruction!
All that's absolute gibberish, obviously. The only reson you are able to make some sense out of it is that you have a human-level understanding of language and that you can place the nonsense strings generated by GT into an appropriate context that can give you some information about their intended meaning. Most however are too garbled to make any sense of them- the second string is the most striking one.
This is not "good enough" (as opposed to perfect). That's just terribly bad. If you think it works for you that's only because you're so good at understanding language by dint of your being human (well- I assume).
Of course human translators get it badly wrong. I've read human translations that are not much better than that nonsense above, to be honest. But a reasonably competent human translator will never make the kind of elementary mistakes that automatic translation does. And those elementary mistakes will always escalate to the point that the meaning of longer passages will deteriorate to gibberish.
As to being able to understand basic signs, posters and menus in Greek- you can do that with a good travel guide also. And the travel guide will not claim to be a breakthrough Artificial Intelligence system, like Google claims for GT.
To expound a bit on poor human translations- when a human translator can't make sense of a passage in the original language, they'll come up with a meaning that makes sense in the context of the translation so-far. So the final translation will make some sense, even if that sense is not an accurate reproduction of the meaning of the original.
When machine translation makes mistakes on the other hand, the translation itself makes no sense. It's not just that its meaning is unconnected to the original; its meaning isn't.
Yes, human translation is better, that's a trivially simple point, but irrelevant for deciding whether machine translation is very good and amazingly useful in the absence of a human translator.
No, the problem that YeGoblynQueenne points out is that when the current state-of-the-art machine translators get it wrong, they get it hilariously, catastrophically, gibberish-wrong.
If a system is 99% right, but the remaining 1% is catastrophically bad, it negates a lot of the part that is actually good. You might end up more confused by the machine translation than you were without it.
Machine translations is one of these systems where 99% good simply isn't good enough.
Sure, I don't disagree with the essence of that, I disagree with how much of the usefulness of the translation that is negated by that 1% (or whatever the right number is). No, you shouldn't make important high-impact decisions on the back of a Google Translate, especially one that's more or less garbled, but there are huge swathes of human communication that are perfectly useful that doesn't even remotely fall into that category.
The problem is that there's a trap here. You and I and every reasonable human can immediately spot when GT produces gibberish, when it's so completely wrong you know it's useless.
But the article had more subtle examples where important nuances are lost in translation, and those are impossible to pick up unless you speak the original language, but we don't speak the original language, because that was the whole point of using GT to start with, right?
Obviously, GT will get better at reducing the gibberish errors, but that just strengthens the trap, because now you will never think it's making any errors, because you lack the knowledge required to know what errors it makes!
> Machine translations is one of these systems where 99% good simply isn't good enough.
I really, really disagree. If 99% lets me understand the gist of an idea, or instructions for something (of course: where there is no risk of harm), I'd take it -- even with the 1% of hilariously catastrophic mistranslations. This means I can go to Japan or Greece and just use Google Translate and be reasonably confident I can manage with a combination of this tool and my human-level understanding of language :) It's definitely good enough for me.
The very first example in the article was one where GT gets the words mostly right, but completely misses the point.
And if you miss the point, then the original text hasn't been translated. It's merely been decoded, this is the entire point of the article. Calling what GT does "translation" is wrong.
Yes, there's absolutely value in decoding signs and texts and simple phrases and directions. It allows you to go to a foreign country and get around without speaking the native language. Great. But a phrasebook can do the exact same thing, except it doesn't claim breakthrough AI machine-consciousness bla bla bla.
I'm not as good with words as Hofstadter, so here's his words from the end of the article that summarizes the problem quite nicely:
"I’ve recently seen bar graphs made by technophiles that claim to represent the “quality” of translations done by humans and by computers, and these graphs depict the latest translation engines as being within striking distance of human-level translation. To me, however, such quantification of the unquantifiable reeks of pseudoscience, or, if you prefer, of nerds trying to mathematize things whose intangible, subtle, artistic nature eludes them. To my mind, Google Translate’s output today ranges all the way from excellent to grotesque, but I can’t quantify my feelings about it. Think of my first example involving “his” and “her” items. The idealess program got nearly all the words right, but despite that slight success, it totally missed the point. How, in such a case, should one “quantify” the quality of the job? The use of scientific-looking bar graphs to represent translation quality is simply an abuse of the external trappings of science."
> [Google Translate] allows you to go to a foreign country and get around without speaking the native language. Great. But a phrasebook can do the exact same thing, except it doesn't claim breakthrough AI machine-consciousness
Wait, who is talking about consciousness? That's a different deal, and let me assure you I'm as skeptical as you appear to be. AI is a buzzword here, don't let it distract you from the effectiveness of the actual techniques.
Also, it stands to reason portable automated translation like your phone can do is not something a phrasebook can do. I can point to a random sign, take a picture of it and have Google Translate give me a helpful if quirky translation. I definitely cannot pull out a phrasebook and hope something matches what I'm seeing in the sign; it will take ages. So it's at least more convenient.
In fact, I've done this with Japanese instructions to assemble a scale model. Granted, here the problem domain was very constrained, and I already sort of guessed what I was trying to do -- and yet, Google Translate was orders of magnitude more helpful than flipping through a phrasebook. Would it be as helpful if I was in completely uncharted waters, where I can make fewer guesses? Probably not. Still, very impressive.
Sure, the example from the article gets mangled in a way that loses all meaning. Yet the author calls it "a trap" for a reason. I stand by my assertion that I'll take the 99% of cases. Yes, they'll have some hilariously funny mistranslations, but they'll be useful anyway in most cases.
I don't know if Google Translate is "good enough", that's an arbitrary scale. What I do know is that it's a lot better than what existed before. Yes, I'm human and know language, but Google Translate exists to serve humans?
The particular paragraph is mangled to the point of being gibberish, yes, and noone with the option to choose between your translation or Google's would give Google's even a second glance. But if I don't, it's that, or a paragraph of text that I can just about identify as being Greek, but which is otherwise, well, Greek to me. But as you concede, with context and human understanding of language, I was able to pick out a few crucial chunks of meaning.
Whether it's good enough depends on what I'm trying to achieve. If I'm judging your skill as a writer and storyteller, then obviously not. But if we hung out with Vasiliki, Mitsos and Maria last year and you put that on Facebook, I'd at least know enough to send my sympathy to Mitsos and have a good chuckle over Vasiliki's romantic endeavours, which would probably be what you intended - - which is pretty much the definition of "good enough".
As for travel guides and Google Translate, as someone who's travelled extensively both pre and post Google Translate, there is nothing in the difference of experience that doesn't qualify as a breakthrough. I have had fully meaningful (if not exactly deeply philosophical) conversations with people entirely mediated by Google Translate. I'm sorry your experiences have been so different from mine.
> The particular paragraph is mangled to the point of being gibberish
You just discovered the poor quality of LSTM generated text. It might come as a surprise to you, but it's legend between AI researchers :-(
If it's not translation (such as in chatbots, image captioning, or computer generated text in general), it's even worse. Unfortunately we can't generate text that makes sense. We can only do it in very specific cases and with lots of errors. Other language models are even worse than LSTMs.
The problem is that automated translators don't have real world experience and can't think causally. Humans can, because we are in possession of a body and live inside a complex world. We can experiment to test our hypothesis, GT only sees text and can't test anything.
> The particular paragraph is mangled to the point of being gibberish
I actually disagree it's gibberish. I could understand enough to make sense of the situation (the part about blades in his leg puzzled me, but I understood about Maria being an ex girlfriend even if the sentence didn't resemble actual human language). This translation is by definition not gibberish.
> But if we hung out with Vasiliki, Mitsos and Maria last year and you put that on Facebook, I'd at least know enough to send my sympathy to Mitsos and have a good chuckle over Vasiliki's romantic endeavours, which would probably be what you intended - - which is pretty much the definition of "good enough".
Btw, the definition above is missing an important point: that all language is, in a sense, gibberish.
What I mean by that is that all the language structures we choose to convey a certain meaning are arbitrary and, by themselves, completely meaningless. We assign meaning to them by convention (and the fact that this convention keeps changing is why we have different languages and why even the "same" language sounds very different a few years down the line). It is when this convention is breached that our ability to understand the intended meaning is compromised.
I'd liken this to adding noise to a signal. The more a passage of text deviates from the conventional structures for the intended meaning, the less the information that can be safely extracted from it. The more the noise, the less likely that 100% of people hearing or reading the utterance will understand 100% of its indended meaning.
So in other words- it's still gibberirsh even if you can understand it. Because maybe you personally can understand it, but any number of other people will not- people with different language abilities, that is.
To draw an analogy- maybe you personally don't need a vaccine for chicken pox because you are immune to it. That doesn't make the vaccine unnecessary. Others don't have your immunity. etc.
No, I disagree with this premise. It is contrary to the definition of gibberish I'm using (and which I linked to). Since we disagree on this fundamental issue, the rest of this conversation is very difficult.
If all language is gibberish, like you say, then it's ok for Google Translate to produce gibberish. (Sorry, that doesn't make sense to me!)
I think what I say is not controversial. Maybe my turn of phrase is not very common. What I say is that language structures don't have inherent meaning. We associate them with meaning in a one-to-one relation, but the structure side of the relation can be anything. This is why it's possible to say the same thing, i.e. convey the same meaning, in many different languages, i.e. relating it to many different structures.
So language utterances mean nothing by themselves. They only have the meaning we assign to them by convention. It is in this sense that I say that those utterances are gibberish.
This might sound weird to you, but if you think about it, most natural language processing tasks basically come down to performing a search for the meaning that is related to a given structure. What makes this search very difficult is that structures are inherently meaningless, so there's no logical reason why a given string might mean something particular. Essentially, we try to teach the NLP system to learn our conventions. But that is very hard to do, because we don't exactly understand how these conventions work, ourselves.
So it's not OK for GT to produce random gibberish- because there is only one set of strings of gibberish that is associated with the intended meaning of a sentence in an original language.
And, I guess that's my point. If GT's search fails to find the right string then it's not doing very well.
It's not controversial to claim everything in the world (physical or imaginary) is essentially meaningless except for the meaning we humans assign it. Then there's the consensus of meaning, etc, etc. Welcome to philosophy! :)
> So it's not OK for GT to produce random gibberish
Thankfully, it doesn't in most cases. If it were random, we wouldn't be able to understand it (please, don't ask anything that starts with "how do you know...?". I know. Let's drop that angle, please!) The chance of all of us understanding roughly the same idea from random gibberish is effectively zero. Hence, it's not random gibberish.
But we don't all understand the same thing from language that doesn't closely follow our agreed rules of language structure. Specifically in our example, not everyone will understand what the mangled translation of my text says.
So this is something to wonder about: why is it that we all speak in a certain way? If we can understand language with unfamiliar structure, why does language follow set patterns? Why don't we all speak any old way we like?
Mind you, I'm not saying I have any answer to this. Because it's actually a major question. You could, for example, observe that people will often break from accepted conventions when speaking or writing and it's hard to even measure the deviation from accepted conventions "in the wild" (as opposed to more formal text or speech). And yet there does seem to be a set of patterns of utterances that we observe when speaking or writing in our respective languages. There are rules, in language, and these rules are observed. Why is that?
Understanding the answer to this question -what makes language, language, rather than random noise- could tremendously boost NLP tasks of all sorts. To be honest, I don't see anyone asking it though. Instead people have become comfortable with the paradox of the random noise (sorry, I insist) generated by computer programs, and treat it as an engineering problem to be solved, rather than an opportunity to understand human language by looking at its difference to machine language (if I may).
>> (please, don't ask anything that starts with "how do you know...?". I know. Let's drop that angle, please!)
I understand if you don't want to spend time discussing this, but I hope you understand that, unless you justify your thinking, you can't expect anyone else to agree with it.
In any case, "how do you know" is the fundamental question you have to ask if you want to advance your knowledge. That's how it all begins- by challengine assumptions (yours or others').
> Specifically in our example, not everyone will understand what the mangled translation of my text says.
Not everyone will understand your hand-made translation, either. Human understanding works by consensus. Someone will be left out of this consensus almost always. It doesn't mean the automated translation is gibberish. If enough people understand the automated translation, then by definition it's not gibberish, because random noise doesn't carry enough information for you to get the idea about Maria, Vassiliki, the hospital or the broken leg.
> I understand if you don't want to spend time discussing this, but I hope you understand that, unless you justify your thinking, you can't expect anyone else to agree with it.
I have justified it. I asked you to drop it because we're running in circles, not because I don't have arguments. Your argument seems solipsistic to me: "but how do you know?" -- I know because it's my experience and because other people agree with me. How do you know your translation is good, anyway? Because it's what you think and because other people agreed with you. Because you managed to successfully -- as assessed by yourself -- communicate the idea you wanted.
The difference is that I can point at the ways that GT fails, in terms of
specific grammatical and semantic errors, whereas you can only report some
examples of people who agree with you. Not trying to be offensive, but your
definition of gibberish is subjective whereas mine is objective and
quantifiable (in one word: ungrammaticality).
To go back to an earlier example of learning to solve mathematical problems
with machine learning- let's call it machine arithmetic. Say we developed a machine arithmetic
system that consistently estimated the sum of 4 + 4 to be 7. That's
consistently close to the correct answer, but it's also consistently not the
correct answer.
So, to pose yet another question: a system trained to solve maths problems who
got it almost right 100% of the time, would be considered pretty rubbish. Why is GT
not? It's computing language, not maths, but its computation is still very
often completely wrong in specific, measurable ways that really leave no
room for subjective interpretation.
> The difference is that I can point at the ways that GT fails, in terms of specific grammatical and semantic errors, whereas you can only report some examples of people who agree with you
What matters for the purpose of human communication is the latter, not the former. Success at communication trumps your grammatical errors (where beauty is not concerned, of course. If you're going to argue automated translation produces ugly results, then you'll hear no argument from me!).
> Not trying to be offensive, but your definition of gibberish is subjective
You're not offensive; you're arbitrary. I provided you with a dictionary definition of gibberish, while your definition seems to be suspiciously ad hoc. You've essentially defined gibberish as "whatever output automated translation currently produces", which makes this debate pointless.
Math and natural language are different. Again, the standards are different, just as in your example with child's language. No, we do not judge a computer's ability to "translate" the same as we judge a young kid's ability to speak. There's nothing surprising about these two standards being different.
Your example with Math is not particularly relevant. However, since your background seems not to be computer science [1], maybe you're unfamiliar with heuristics: in the context of computer science, a heuristic is a technique where you take an optimization problem (e.g. "finding the minimum of something", "the shortest route to somewhere", etc) which is computationally untractable and provide a
"good enough" answer which can be computed. The definition of what is "good enough" is, of course, relevant to the application of a particular heuristic in a specific context. In this sense, for some heuristic and its application, maybe 7 is a "good enough" answer even though the right answer is 8.
[1] This was written before I read in another reply of yours that you claim to be working on a PhD in Machine Learning. Fair enough! But even if you're familiar with heuristics, I think my point stands.
>> What matters for the purpose of human communication is the latter, not the
former. Success at communication trumps your grammatical errors (...)
But grammatical errors cause communication to fail. Because they introduce
(measurable) noise to the signal, which obviously makes it harder to extract
information from it.
Btw, grammaticality is another uncontroversial measure of the performance of
NLP systems that are required to generate language (like machine translation
does). There is some theory behind the idea that well-formed sentences are
inherently more intelligible, that will take a lot more to overturn than "it
works for me".
To draw yet another analogy- homeopathy "works" for some people: they feel
better when they take its concoctions. That doesn't make homeopathy
objectively useful.
>> in the context of computer science, a heuristic is a technique where you take
an optimization problem (e.g. "finding the minimum of something", "the
shortest route to somewhere", etc) which is computationally untractable and
provide a "good enough" answer which can be computed.
Yes, looking at machine translation as a heuristic is a good point. Like I say
in other comments, the problem of machine translation has no good definition.
That's because it's very difficult to define what is a "good translation" in a
formal and principled manner. Hence, we're left with heuristics like "how
close this automatic translation is to this human translation" (where the
human translation is whatever happens to be available, rather than something
chosen for its quality or other attributes).
This is actually exactly what machine translation systems do. They are made to
optimise a measure of error over examples of human translation. So GT is
actually designed to do what you say is not important, produce grammatically
and syntactically correct language, like its examples are.
In that, it fails. It may achieve your goal of "communication" but that is not
really a heuristic, it's more of a quality that you, personally (and obviously
other people too) assign some value to. But it's far from a universally
accepted measure of machine translation quality. You might say that
"communication" is a business goal whereas matching human translations is an
engineering goal.
As a P.S., I'm not "claiming" to be working on a PhD. The information is in my
profile with links to my linkedin and github :)
> But grammatical errors cause communication to fail. Because they introduce (measurable) noise to the signal, which obviously makes it harder to extract information from it.
But when communication doesn't fail, grammatical errors don't matter that much. Because communication success is its own evidence (cue your "but how do you know [...]?" -- please don't. We've been down that road and your argument is unconvincing and solipsistic).
You say grammar correctness is another uncontroversial measure of NLP, but nobody was arguing against this. Of course it is. You're shifting the goalpost. Grammar matters. You can tell when something is ungrammatical but whether this affects communication is another issue. It's also a matter of degree: something can be so ungrammatical it conveys no meaning; and something can be ungrammatical enough to sound "bad", but still convey its intended meaning.
> Hence, we're left with heuristics like "how close this automatic translation is to this human translation"
That's one heuristic, sure, and a pretty good one! Not the only one, though. In any case, you missed the point that your Math example was not very good, because it ignored that in a heuristic, what you proposed as a bad "translation" could actually be a good one. In the realm of human language translations, it's never a case of 4+4=8.
> As a P.S., I'm not "claiming" to be working on a PhD. The information is in my profile with links to my linkedin and github :)
My mistake then! No offense intended.
PS: one more thing: I've looked at your comments history and I actually agree with your skepticism about AI. I think it's overhyped, especially by people with no background and no formal understanding about it -- like Eliezer, who you rightfully mock in your profile, and many other Prophets of the AI Apocalypse (or AI Utopia, depending on their mindset). So consider me a skeptic as well. We're more in agreement that it seems! I just don't think the kind of translation we're talking about is anywhere close to General AI or "understanding" of any form, and I can get excited about a "limited breakthrough" even though it's not actually about AI :)
Well, alright then. Let's agree to disagree so I won't say anything else. This thread's text is already pushing very aggressively towards the right side of my browser window anyway.
>> No offense intended.
... and no offense taken, absolutely.
We'll probably get the chance to argue again in the future, anyway, given that you're interested in the same subjects as I am :)
Because non-gibberish doesn't mean "the best/most common way to communicate". If you speak in English like Yoda, the character from Star Wars, you're perfectly intelligible -- hence, you're not speaking gibberish -- even though (almost) no-one speaks that way in real life.
Gibberish means nonsense. If most people understand you, it's not nonsense.
You're right: I do not know for sure that most people understand Google Translate's translations (with more or less effort). I assume they do because I do, and because other people I know also do.
>> I have had fully meaningful (if not exactly deeply philosophical) conversations
Any conversation can easily become "philosophical" when Google Translate messes up and inserts random garbage instead of intended meaning, and people, instead of dismissing translation as garbage, decide to seek some deeper meaning in it :)
> "The only reason you are able to make some sense out of it is that you have a human-level understanding of language and that you can place the nonsense strings generated by GT"
I think you're understating the quality of the automated translation here. It's by definition not nonsense if I can derive a pretty accurate picture of the situation by reading it. While flawed, it's not random gibberish. If it were random, I -- not knowing how to read Greek -- wouldn't be able to understand about someone being in the hospital because of a broken leg, about some friends visiting him, and about the social faux pas of the two girls being in the same room as the patient. Yes, some of the mistranslations are funny, but they are not nonsense.
If an automated translation is good enough for a human-level understanding of language to parse and derive a decently accurate picture out of it, while not understanding a single word of the original text, then that's an impressive achievement!
By the way, those of us who remember the initial results of early Google Translate should be impressed. It was truly, really unusable. It has gone a really long way, to the point I consider using it in places where I really cannot understand a single word or read the script... like in Japan, for example.
To restate my previous comment, the fact that you can understand a badly mangled phrase is not evidence that it's not gibberish- it's a testament to your ability to glean meaning even from gibberish.
Is a machine that produces gibberish that's not too mangled to understand better than having no translation at all? It certainly is. However, Google's hype machine will not stop there- they consistently overstate the ability of their systems, by a very large margin.
If Google was saying "look, we have a machine translation system, it's a bit shit but it gets the job done" that'd be fine. But they don't say that. They pretend their system actually solves machine translation, for real, like AlphaZero solved Go. You just have to watch the triumphant press releases that come out of Google.
> They pretend their system actually solves machine translation, for real, like AlphaZero solved Go.
The post you linked to makes no such claim. The strongest claim to, well, anything, is that it produces "reasonable" translations between language pairs it has never seen.
This inspired us to ask the following question: Can we translate between a language pair which the system has never seen before? An example of this would be translations between Korean and Japanese where Korean⇄Japanese examples were not shown to the system. Impressively, the answer is yes — it can generate reasonable Korean⇄Japanese translations, even though it has never been taught to do so. We call this “zero-shot” translation, shown by the yellow dotted lines in the animation. To the best of our knowledge, this is the first time this type of transfer learning has worked in Machine Translation.
Because what they claim to do is really out there- it is something that not even humans can do. If they could actually do it then they would have solved machine translation for real.
And they haven't - like I say above, they seem to have misinterpreted a model of English-to-many (other languages) as an interlingua. And it's not even a very good model of English-to-many.
So the difference between hype and reality is astronomic and I feel very well justified to say what I say.
I disagree. By definition, if it's gibberish no amount of human-level understanding of language will be able to glean meaning from it. You cannot extract meaning from random noise. This is better than random noise; in fact it's way better than what Google Translate used to manage a decade ago.
It's not gibberish. It's just very bad quality. But it helps me extract the meaning, which is not that far from what I do with some of the emails I receive in my day job -- trust me!
This is absurd. You keep insisting on perfection or nothing at all. There is such a thing as incremental progress.
Google Translate is supremely useful in the absence of a better alternative. Nobody is contesting that human translation is better than Google Translate, but a human translator isn't always available. Google Translate is better than nothing at all.
A child's first many, many attempts at language are absolute gibberish, then slowly individual words, then combinations of words, then gradually more complex sentences etc -- gradually over several years. When you child first says "mama", the common response is excitement, not berating the child and anyone excited that "mama" isn't Wordsworth, and asking if just saying "mama" is acceptable, why aren't we all just communicating using single words all the time?
A child being able to express itself at all using language is better than it not being able to, and cause for celebration, and so too it is with Google Translate.
>> You keep insisting on perfection or nothing at all. There is such a thing as incremental progress.
I think we're talking at cross-purposes here a bit. I don't really care whether GT is useful as a tool. I care about whether the translations it offers are good translations or not. I think you're basically saying that, if it's good enough for you, then it's a good translation. However, this is not a good measure of the quality of a translation! To begin with, what's good for you is a sujective measure. Further, we wouldn't accept "it's good enough for me" for any other translation than a machine translation- and it's accepted for GT only because we understand the limitations of machine translation systems.
So basically my pointing out the limitations of machine translation systems has nothing to do with whether it's useful to you. It's an attempt to explain why the results of machine translation are objectively bad.
I'll agree if you want that it's very difficult to get to such an objective measure of goodness or badness. But that's actually a reason why it's so hard to do machine translation properly: because it's next to impossible to evaluate machine translation objectively, which in turn makes for a very poorly defined task.
So you can feel justified to say that GT is OK for you, because you can understand it. Others can say that it sucks because they can't. And Google can say it's a breakthrough in AI because it scores high on their formal tests, evaluating using BLEU score (or something equally arbitrary).
But none of those tells us how good or bad GT translation really is. So we're left with what Hofstadter does (and what I did here): eyballing it. And, eyballing it, you can see that it's making elementary, egregious mistakes that are not justified by the hype surrounding GT as a product.
Btw, we can expect a child to get better at using language because most children do. So far, no machine translation system has got a lot better than GT- and I don't even agree that GT has gotten any better over the years, or that it's even any better than earlier systems. It just happens to be Google's system and they advertise it as the best ever- but they can't actually, you know, prove this.
> if your child was speaking in such a manner you would be very, very concerned. But with GT it's OK? Why?
Because if, past a certain age, my child keeps talking this way, it's a sign of development problems. Google Translate is an algorithm and I don't worry about its mental health. The standards for software/machines and human children are different and I find nothing surprising about this fact.
I don't think anyone is claiming Google Translate "is OK" in the sense of "we're done here, we've done all we could, let's stop thinking about automated translation and declare it mission accomplished".
My point about child speech is that the kind of utterances produced by GT would not be acceptable as correct speech if they came from a child, let alone an adult. We'd recognise them as imperfect. Coming from GT though they're considered "good".
Two other comments accuse me of letting perfect be the enemy of good. But for this to happen, GT translations must be "good" in the first place. Well, they're not. They are often terrible.
I don't understand why this criticism is even controversial. Terrible won't become good if we don't point out that it is, in fact, terrible.
Plus, I'm very concerned about promoting commercial products as examples of technological and scientific progress. Of course Google has an incentive to claim its translation service is great. Why does anyone else?
> My point about child speech is that the kind of utterances produced by GT would not be acceptable as correct speech if they came from a child, let alone an adult. We'd recognise them as imperfect. Coming from GT though they're considered "good".
Because the standards for automated translation, something that didn't exist a few decades ago, are different from the standards of human speech and translation? No-one is saying Google Translate is on par or anywhere near human translation.
> Terrible won't become good if we don't point out that it is, in fact, terrible.
Yes, but you see complacency where there is none. The engineers who achieved this are rightfully proud about it, but I doubt they think this is the end of the line for automated translation.
It's helpful to say "this is flawed here and there, you can do better!", but your attitude seems needlessly antagonistic. Frankly, it sounds to me as if you feel threatened by this development. You shouldn't. As you point out, this is miles behind actual human translation in subtlety. Human-level understanding of speech (with double meanings, play on words, puns, understatements, etc) may very well be impossible for software to achieve; in fact, I think it's forever out of reach. But that's not the goal of automated translation; it's merely meant to be a useful tool when there are no human translators available.
> Plus, I'm very concerned about promoting commercial products as examples of technological and scientific progress. Of course Google has an incentive to claim its translation service is great. Why does anyone else?
Because they are definitely examples of technological progress and we can feel excited about it, and hope it gets even better! I couldn't care less about the Google brand. I understand your concern and I'm not excited about Google-the-business.
>> No-one is saying Google Translate is on par or anywhere near human translation.
Well, they kind of do. Like I say elsewhere, the common measure of machine
translation performance is a family of metrics like BLEU or ROUGE that basically
compare machine translation to human translation. So when a research team
claims strong performance in machine translation, they're really saying that
their systems can do the kind of translation that humans can do, in a very
literal sense (of similarity between tokens or n-grams). Besides, like
Hofstadter says, Google has often claimed its NLP systems (for example, its NL
Parser, Parser Mc Parseface) approach or outdo human performance.
Google is quite unrealistic in the way it promotes its technology and claiming
it does better than humans is its bread and butter in this regard.
>> Frankly, it sounds to me as if you feel threatened by this development.
I don't have any reason to feel threatened :) I'm a machine learning PhD
researcher with some hands-on experience of NLP (though not machine
translation specifically- I've only theoretical knowledge of it). I have a
background in foreign languages and translation, but not professionally.
If I can summarise my concerns, it's not that I'm worried that machine
translation will become so good it will take the jobs of human translators.
I'm worried that we will keep replacing human workers with bad AI and end up
making our lives a lot worse in the process.
Would you be willing to try http://www.deepl.com/ and say if it is better? Google translate is not SOTA because it has huge volume and they can't afford to use expensive models for free.
Not enough training data for greek language it seems. Here's russian version.
"Я и Вассилики пошли проведать Митсоса в больнице. Он сломал ногу и ему поставили металлические вставки. Вассилики, которая немного влюблена в него, принесла ему шоколада и цветов как глупая девочка из романтического сериала. Я пошла только за компанию. И знаете что было хуже всего? Не только его мать и отец были там, но ещё и Мария - его бывшая с прошлого года. Ужас!"
"I and the Vassiliki went to see Mitsos in the hospital. He broke his leg and put metal inserts. Vassiliki, who is a bit in love with him, brought him chocolate and flowers as a stupid girl from a romantic TV series. I went only for the company. And you know what was the worst? Not only his mother and father were there, but also Maria - his ex from last year. Horror!"
I don't understand Russian, but it would be interesting to hear what you think of the Greek-to-Russian translation. If I'm right about it, this will go through English and the result will make it obvious. However, since I don't know Russian I don't really know what to expect.
Sub par quality of the translation is evident even with no understanding of the source. But to be fair I understood even less of the Greek text, when I tried to use online dictionaries.
I used to know a couple of people working on rules-based translation, but that approach seems to have been sidelined by the early successes of statistical methods. I hope that work has continued in the background, not only for its own sake, but because statistical methods may have reached a point of diminishing returns.
The lack of understanding in statistical machine translation is a counter-argument I use whenever someone claims that its apparent success is a sign that the AI singularity is imminent, but I have to admit that the recent progress in statistical AI has led me to wonder if, just maybe, these methods are closing in on the fundamental methods of understanding. Perhaps John von Neumann's famous quip "Young man, in mathematics you don't understand things. You just get used to them" is actually generally true.
Nice examples. But the problem with "swallow" and similar lies somewhere else. It's that translating a single word into a single word is a ridiculously stupid idea. As a person whose mother tongue is not English, I occasionally stumble on a word that I need to translate. GT is utterly useless for me. A random free online dictionary gives me 20 different translations for "swallow".
I am honestly surprised that a company with such talent creates a tool that is this useless; it's literally about doing something like this:
1) Check if the word or phrase is in a dictionary. If it is, return all results from the dictionary.
2) If it's not, try something smarter.
But no, this is not Artificial Intelligence enough! /s
At least Google Translate is a bit better than Baidu Translate which at some point decided that my name is the English translation of 扒饭 (grilled rice).
Wow, that's awesome, the only way it could be more surreal is if you had been holding the menu in front of you!
"扒饭" isn't, on its own, an actual Chinese word. The "扒" is basically a modifier that comes after a type of meat and indicates that it is a "steak" or "cutlet".
So your menu item "美式杂扒饭" should translate as "American-style" (美式) "mixed grill" (杂扒) "on rice" (饭)... but the machine translation is getting the word boundaries wrong and translating it as "American-style" (美式) "mixed" (杂) "larrysalibra" (扒饭). Fascinating.
The really crazy thing is how I found out about it. A friend in Austin's friend who I don't know was apparently at that restaurant in Shenzhen, ~10 miles north of where I live in Hong Kong, traveling on business, saw the menu, recognized my name from somewhere, told the friend in Austin who then passed on the message and pic to me. Small world.
China's a big country - makes me wonder what other menus I'm on!
The author is Douglas Hofstadter, author of Gödel, Escher, Bach.[a]
In this article, he shows with concrete examples how Google Translate falls short, and then offers two criticisms:
* Feeding more data to current models won't bring them any closer to understanding, since understanding involves having ideas (including ideas about the state of the world), and this lack of ideas is the root of all the problems for machine translation today. His examples are powerful evidence of this limitation of current state-of-the-art machine translation systems.
* Current machine translation systems make no attempt to go beyond the surface level of words and phrases. These systems merely discover statistical regularities that relate words to other words at multiple, hierarchical levels of composition. In Hofstadter's words, "there's no attempt to create internal structures that could be thought of as ideas, images, memories, or experiences. Such mental etherea are still far too elusive to deal with computationally." He is right.
That said, AI researchers are aware of these limitations, and are exploring possible ways to overcome them. An early, crude example of such research is the multi-modal model of "One Model to Learn Them All" (https://arxiv.org/abs/1706.05137), which is trained to learn to perform multiple image-recognition, language-translation, image-captioning, speech-recognition, and language-parsing tasks at the same time, using representations that are shared by all tasks.
While these early research efforts fall far short of the kind of "understanding of the world" Hofstadter shows is necessary for human-level language translation, it's encouraging to see AI researchers actively looking for ways to move beyond the mere discovery of 'hierarchical statistical regularities' that relate words to other words.
I was under the impression that modern word embedding techniques have emergent properties that start to approach conceptual understanding. The trick is figuring out how to leverage that property properly.
I believe Google Brain has a demo where they show translation between two languages without having any corpus specifically linking the two (i.e. the system learned English/Japanese translations and English/Greek translations and then translated Greek/Japanese) with a reasonable degree of accuracy. This seems to me to hint at a more conceptual understanding as opposed to a simple pattern matching activity.
In either case, I 100% agree that the multi-modal model is the path forward. It feels like there's a lot of low hanging fruit in the area of model ensambles.
I sell things on Amazon European platforms (UK FR DE ES IT) and they receive reviews.
I only speak English and French and therefore for all other languages I try to understand the reviews using Google Translate.
The result is never good. Sometimes it's barely intelligible; many times it's not really. (For some reason, GT is incapable of translating Italian; by which I mean: the resulting translation gives you no idea whatsoever about the original meaning.)
It really makes one wonder if the hype/fear about AI is maybe misplaced.
Google is a self-described "AI company", with all the money in the world, and staffed with the best people, and access to the most data, and this is all they can come up with?
The only explanation is that the problem is simply too hard.
It already works very well, especially after its improvement in the recent years after using neural networks. I've frequently been surprised by the quality of the translation it provides to Japanese sentences, not to mention between German and English. The test cases I put in in some other languages (e.g. Chinese) are really promising too.
Machine translation is a really hard problem. The messiness of language as a system, the importance of context in daily conversation etc. all play a part. Another layer of complexity is the gap between everyday usage and official, written form, which is also being tackled by researchers. You have to put this thing into perspective. Many of the old rule-based/Chomskyan software have been simply unusable for decades. New statistical approaches have been in use for barely 10 years, and industrial deep learning less than half a decade. There are still much more to come. The hype IMO is well justified.
Well, that's really not my experience, nor is it the experience of the author of the article. The examples given in the article speak for themselves, really.
Really? I was actually surprised of how well Google translates Italian in most cases (I'm Italian).
Most of the times, I encounter major translation issues when the original sentence is barely intelligible in the original language. Other times there's a small mistake which could be understood by context in the original language but it's completely lost once translated — like it happens when you try to translate jokes and puns or when you miss a comma.
People generally don't write correctly, so Google needs a lot of luck to understand that "Let's eat grandma!" doesn't actually translate to "Mangiamo la nonna"
---
I translated this whole comment for the sake of it and the result is extremely good. Not completely natural but well beyond understandable.
I translate FROM Italian, not to it, and, well, most of the times, I don't understand the result.
(It's likely Google Translate is better at going from English to another language, since it seems it uses English as a pivot. In fact that's how I use it if I want to write something in, say, German; I write in English, and check the French translation, and rephrase and rephrase, and don't accept the German version until the French translation is acceptable).
Going back to Italian, here's a small example: Non potevano dirlo solo gli adulti?, which really means, I think, "Isn't it something only adults would say?" is translated by Google in English as:
Could not only adults tell it?
which doesn't seem to mean anything, and in French as:
Les adultes ne pourraient-ils pas le dire?
which means something, but a different thing from the English translation, and, I think, not the correct interpretation of the original Italian one.
Maybe the problem is not too hard, just that statistical models are not the right way to solve it. Actually it seems like it's THE problem that exemplifies the difference between partial problems that are suitable for stats (bayesian spam filters) and the "true" AI.
Also using English as an intermediate step is a no-no. I doubt anything will improve while there isn't a competitor that tries a fresh approach and gets better results.
There was a time when a lot of people in the irc added an "x" to their nicks in the fashion of unix, linux, aix, hp-ux... I can't help to make the association every time :)
Ho humm. Of course he is right but is that enough? This reminded me of a quote from Doctorow's Microsoft Research DRM talk which I remember well because I translated (hah) it to Hungarian as my last act as a Hungarian journalist -- I sort of resurrected myself as one because by that point I haven't written in years.
> This is the overweening characteristic of every single successful
new medium: it is true to itself. The Luther Bible didn't
succeed on the axes that made a hand-copied monk Bible valuable:
they were ugly, they weren't in Church Latin, they weren't read
aloud by someone who could interpret it for his lay audience,
they didn't represent years of devoted-with-a-capital-D labor by
someone who had given his life over to God. The thing that made
the Luther Bible a success was its scalability: it was more
popular because it was more proliferate: all success factors for
a new medium pale beside its profligacy. The most successful
organisms on earth are those that reproduce the most: bugs and
bacteria, nematodes and virii. Reproduction is the best of all
survival strategies.
I think Doctorow's point is dramatically different here. He's talking about a change in medium and specifically arguing against the claims about the original experience offered by DRM controlled media (e.g., theatres, ebooks, etc).
The article's point is more about while Google Translate has its impressive moments, it needs to be understood for what it is - the most basic translation services.
If you've ever had to rely on Google Translate (or any automated translate service) you'll know what I mean by "basic translation services". It's great if you want to say "I want to order Fish and Rice" or "Excuse me, where is the restroom". The moment you try to get past such complexities is where the automated translations just can't get a good grip on human languages. Languages from the same families naturally are a bit easier (English to Spanish, for example usually can allow for more complexities), but you still lose a lot simply because it's a 1:1 translation from the box on the left/top to the box on the right/bottom, which doesn't happen.
I'm learning Russian (in Russia no less) and one of the hardest things for me as a native english speaker is avoiding the use of "to be", which is almost never used in Russian. There are other oddities as well regarding prepositions in Russian that are handled differently than English (my poor teacher still can't satisfactorily explain to me why stuff is "on the kitchen", but it's "in the ___" for other rooms) (в гостинной versus на кухне). Now, surprisingly Google translate gets these two right, but other such preposition specialties it bungles completely, and I've had more than my fair share of confused looks when I have to resort to using translate for more complex sentences. Especially when it comes to things like words of motion, for which there are tons of different words depending on how you're going.
It's good that it's accessible, but when the reverse happens (russian to english), for me it requires a fairly fast sed operation to fix Google Translate's choices for words or just some additions to help get what the person is wanting.
Edit: Addendum to the main point, I can see that there may be applicability in that Translate Tools will allow for pidgin languages to crop up if it receives wide spread usage in real world settings, but I'd be curious to see from a large sample how many people use the conversation style features. I'm sure there's plenty of anecdata on the subject from those who have to use it (myself included), but I wonder if it seems as well used if you increase the sample size out further. Globally, people still pretty much stay at home and/or know enough english to get by.
There is a new online translator, http://deepl.com, which relies on deep learning techniques and provides higher (semantic) quality and accuracy of translation. Previously I had quite positive experience with http://translate.yandex.com (but I had to manually compare and combine their results with google translate).
> Human: After the defeat, many professors with Pan-Germanistic leanings, who by that time constituted the majority of the faculty, considered it pretty much their duty to protect the institutions of higher learning from “undesirables.” The most likely to be dismissed were young scholars who had not yet earned the right to teach university classes. As for female scholars, well, they had no place in the system at all; nothing was clearer than that.
> Google: After the lost war, many German-National professors, meanwhile the majority in the faculty, saw themselves as their duty to keep the universities from the “odd”; Young scientists were most vulnerable before their habilitation. And scientists did not question anyway; There were few of them.
> DeepL: After the lost war, many German national professors, now the majority of the faculty, regarded it as their duty to protect the universities from the "oddities"; the most defenceless were young scientists from their habilitation. And women scientists were not questioned anyway; there was little agreement on a few things.
I'd say the DeepL translation is slightly better. It still misses most of the points.
I just tried the following, to German, and got the predictable result, which is completely wrong. Until the proper semantic parts of the text are identified, these sorts of mistakes prevent statistical methods from being trustworthy:
The translating software can't distinguish between 'sailing ships' the noun, and 'sailing' (verb) 'ships' (noun) the act. The latter is what is probably intended. It is also missing chances of using idiomatic expressions in the translations.
Well, 'is' is singular so it can’t be the correct verb associated to 'sailing ships' if the latter means multiple sailing ships rather than the activity of sailing ships.
I like that I can clikc a word in the target box and choose an alternative translation while the system updates the result accordingly. Previously, I had to play with different translation alternatives manually in Google or Yandex translate. I think deepl also learns from such user choices.
Trying the examples, for the French translation it just doesn't make the last plural mistake, correctly translating "la sienne" (but still missing the subtleties)
The German example is "better" I think, it kinda messes up but in a more predictable way.
"the most defenceless were young scientists from (before) their habilitation"
Also the last phrase is very, very tricky, it actually means "there were only few things people agreed on more" and it was translated as "there was little agreement on a few things."
At home, they have everything in duplicate. There is her own car and his own car, her own towels and his own towels, her own library and his own library.
I use deepl for work (native English speaker with fluent German). The EN-DE translations are brilliant IF you keep sentence construction simple. It also helps to write in a German 'style'.
I think this is what the article would refer to as "writing German using English symbols". You're doing the semantic remapping yourself, such that the system can map from one language directly to the other and end up with a good result.
Having used Google Translate to translate at least 6 Russian academic papers in full and about 40% of a Russian dissertation (along with a far larger number of partial translations of papers), I can say that Google Translate works decently for this task. The result won't win any awards for elegance, but it's by large intelligible. I submitted many corrections over the past years and that seems to have noticeably improved the quality, at the very least reducing the number of gibberish sentences. My contributions are limited to fluid dynamics, so perhaps they won't generalize, but I am very happy with the results.
Google in particular had problems with technical phrases which it translated literally, where the literal translation does not correspond to the equivalent English phrase. In one case there was no equivalent English phrase and Google Translate returned a phrase that seemed like it had a Latin etymology. After reverse engineering the word based on the Latin, I recognized the concept and found it interesting that there was a Russian word for this. If I recall I added a footnote explaining the word. Anyway, I doubt your average human translator without subject knowledge could do this. So I don't blame Google Translate for this too much.
I have a list of problem sentences that I'll take to a human translator some time in the future. Machine translation has been convenient but does not yet replace human translation even when one sets a fairly low bar as I do.
The articles absolutely contained information not available in English. I would not have translated the articles in full if they had, as my time is limited. My specific field is the breakup of liquid jets into droplets. As an example, I found a paper that applied the maximum entropy principle to the prediction of droplet size distributions in 1938 (by V. Ya. Natazon), almost 50 years before the approach was popularized for droplet size in the west (and before Jaynes). I actually was not attracted to the paper for that, though. The specific constraint used involved a measure of the strength of the turbulence, and that was what attracted me as my own research was going in that direction. The constraint would be regarded as novel if it were published in a western journal today. I am working on a conference paper right now which cites this.
As for how I find the articles, there are many approaches. I found the Natanzon paper by reading an old English translation of a Russian textbook. Textbooks can be particularly fruitful in terms of foreign references. I also found bibliographies useful to find citations along with English abstracts (though few are recent). Going backwards through the citation network for a particular foreign paper you find interesting can also return good papers.
It's also worth noting that in the Cold War era many translations were produced, though they may be hard to track down today. I only translated articles where no previous translation existed best I could tell. I could write more on identifying whether an article was translated and tracking down the translation if you are interested.
Hofstadter's strength is his ability to stay focused on the highest level cognition problem. His theory is that analogies are the core of human cognition. His main work revolves around discovering how extensive human ability to think using analogies is.
I especially like his attempts to understand and capture the high level cognition in simple toy problems. His `copycat` and `Letter Spirit` programs illuminate the problem and his thinking in very clever way.
The current AI and ML research is building things bottom up and there is still significant gap before we reach the high level cognition that Hofstadter is interested in. What is the representation that binds low level and high level cognition and allows high level fluid concepts to be used as analogies from one domain to next with just one or two examples? Style transfer, variational autoencoders and transfer learning are very limited in this regard.
Challenges for deep learning:
* Deep Letter Spirit. Show 1-3 examples of lowercase letters of the roman alphabet in some font and have an algorithm that understands the style and completes the rest of the alphabet in the same style.
In defense of Google Translate, I will have to point out two constraints of machine translation that human translation usually do not face.
1. Machine translation lacks a direct conception of the physical world: it only understands the "grammar" imposed by physical constraints indirectly through digitized verbal corpora and hand-constructed parameters.
2. Machine translation does not have the luxury of understanding their target audience's domain knowledge. Much of the jargon Google Translate does not understand comes from very specific situations that few people generally experience, e.g. Pan-Germanic. Normally, if such words were used in a news article, the journalist should have to spend a sentence or two describing what is meant by using those words. If Google Translate was tuned to favor translating jargon into jargon, it is likely that its translations would contain much jargon from various domains of experience and be very difficult to read in general.
I heartily recommend Umberto Eco's text "Experiences in Translation" (full text here, a little long: https://archive.org/stream/UmbertoEcoEXPERIENCESINTRANSLATIO...), which perfectly describes the difficulty (and I'd say beauty) of the act of translation.
> In this book Umberto Eco argues that translation is not about comparing two languages, but about the interpretation of a text in two different languages, thus involving a shift between cultures.
The above short presentation of the book says it all, basically, but surprisingly enough is really hard to understand for lost of technical people who only think that translation is just "de-coding", as Hofstadter explains. We won't have proper translations until AGI is here.
> although Google Translate is extremely useful (and I use it all the time), it is true that it does not usually match the skills of the best human translators
I think it is safe to say that Google Translate almost never macthes the skills of human translators except for rare trivial cases. As a rule it's rubbish, and sometimes as an exception it can be OK. Nobody argues against its usefulness, especially considering it's free and that it's pretty much the only tool we all use to understand things written in other languages. Better than nothing for sure.
Further, the whole analogy with flying and birds completely misses the point. Airplanes do satisfactory job at transporting humans and cargo, they are not required to fly like birds or mate with birds. We have invented what seems to be a more efficient method of flying using engines rather than flapping. All in all, airplanes are not birds and were never meant to be.
Similarly, in my view the arguments regarding machine translation come down to two things today: firstly whether it's useful or not, and secondly wether ML improves translation or not. My answers are: (1) weeeell, yeah, it's kinda useful in a better-than-nothing kind of a way and (2) doesn't seem like a significant improvement to me, if at all.
> I think it is safe to say that Google Translate almost never macthes the skills of human translators except for rare trivial cases. As a rule it's rubbish, and sometimes as an exception it can be OK.
Depends on the language pair.
Depends on the human translator. I met translators who wouldn't compete against 1990s Babelfish / SYSTRAN well. They still had some business.
I think it is unfair and even wrong to compare GT to the job of professional real time translation, because the latter is probably one of the most challanging jobs that I know of, next to astronauts and film stuntmen. Instead, what we mean most of the time is "this piece of text in the language I speak fluently that came out of Google Trabslate reads like nonsense to me, but I kinda get it (though sometimes I don't)". Meaning, it's way below my standards of quality for a language I know very well.
Google translate has lots of disadvantages but it is at least fast. Professional written translators manage a few tens of words a minute; professional simultaneous translators manage to keep up most of the time but are occasionally defeated by languages with very different sentence structures, and require frequent breaks.
This amazingly misses the point. Shallit's arguments with birds are the kind of arguments a student has before taking an AI class and feels rather smart about it.
Even though airplanes are different from birds, airplanes still need to follow the laws of physics and aerodynamics. Birds too need follow the laws of physics.
With intelligence, no one knows what the laws are. That is Hofstader's point.
I think he’s (deliberately? Give him the benefit of the doubt) misinterpreted Hofstadter’s use of the words “processing text”.
What he means, which seemed quite obvious to me, is that the machine is not reading text, building up a semantic interpretation of the sentence in the way a human does. That’s because a neural net is a simple pattern recognition machine that does not work in the same way as the brain. It doesn’t have the immense life experience to draw upon (probably relatively easy to fix) but more importantly it doesn’t have a concept of what a sentence actually means.
A neural network doesn’t have an understanding of what “double” means. It just pattern matches a translation.
I think there’s some serious symbolic reasoning going on in the brain, which neural nets don’t yet perform. It all feels like shallow syntactic matching right now, rather than semantic reasoning.
Unfortunately I don’t know how the brain works, so you end up in an absurd argument where people say “the brain is just a neural net” and it’s impossible to completely refute their claims, just as if someone said “the brain is a very large lookup table”. Well, from what I see it does seem that the brain is much smarter than that, but I can’t be certain without knowing what that extra kicker is. So whilst such an assertion seems terribly simplistic and self-evidently insufficient, it is difficult to argue with.
“The brain is just a neural net” plus real life experience, in fact years and years of experience. Pretty much every example that automated translation gets wrong is when a real life context is required. Our language is not just a sequence of words and sentences, it almost always implies some contextual klnowledge. Where two humans have siginficantly different backgrounds they may have difficulty understanding each other for the same reason. A total lack of real life experience on one of the sides makes it even worse: it produces barely comprehensible near-nonsense.
One problem with any attempt to map human intelligence to different types of artificial intelligence is that we only have a very precious few such types of AI, so there may be any number of things we're missing.
It's a case of not having the right analogies. We liken various bodily systems to machines: the heart is a pump, the lungs are funnels, the kidneys are filters, etc. These work up to some point because we understand both sides of the analogy well enough- we understand how the heart works, to the extent that it works like a pump and we understand how pumps work, etc.
But with intelligence we don't have this luxury. We don't understand how intelligence works, yet we draw an analogy not just with computers ("the brain is a computational device") but with specific types of computer programs. However, there are, literally, an infinite number of different computer programs and an unknown number of them could produce results similar, or even identical, to our intelligence.
Of course understanding intelligence is basically coming up with a good model of it. But that must be preceded by a good understanding of how intelligence works, which we currently don't have. Instead, what some researchers do, is that they take their arbitrarily chosen favourite AI model and try to find a way to argue that it's "like" human intelligence.
Neural networks are particularly guilty of this sort of thing. The whole idea of connectionism is to mimic the way the brain does intelligence, however we don't know what that is, so we've just come up with a complex machine that can optimise systems of functions, instead (I mean the set of neural network architectures). Then, when this machine turned out to be good at doing what it was designed for, optimising systems of functions, we claimed this as proof that it's actually doing what the brain does. That's a very circular way of thinking.
> When Hofstadter says "There's no fundamental reason that machines might not someday succeed smashingly in translating jokes, puns, screenplays, novels, poems, and, of course, essays like this one. But all that will come about only when machines are as filled with ideas, emotions, and experiences as human beings are", that is just an assertion. I can translate passages about war even though I've never been in a war. I can translate a novel written by a woman even though I'm not a woman.
When refuting his claims, he also makes some errors. Hofstadter has some very good reasons to justify what he says, whereas Shallit's argument is more or less "we can do slightly better translations now than before, so there is no reason they couldn't be better" - whereas the whole point of Hofstadter is that it's impossible to do exactly using current methods.
I understand both views and I think Shallit might have a point, but he doesn't justify it well. Simply by using statistical methods you can achieve surprisingly good results, although the error ratio is still too high IMO. What we already can do is to produce relatively good translations of texts belonging to well defined domains, such as legal documents. But in order to do it well with all domains, we'd need to find a good method of passing on the necessary context information to the translation engine.
Imagine translating subtitles of a movie. It's absurd to expect the machine will produce better results than a human as it's lacking visual cues. However, if we manage to transmit this information to the translation machine (via Computer Vision, audio profiling etc.), it can get much better results. It's very difficult to expect good results could be obtained just by training the neural networks based on previous movies. Yet, this is what Shallit seems to argue.
I was expecting this reply to be like "yes translation is not yet perfect but we have some ideas how to fix this, and the solution is different than what Hofstadter thinks". Instead it was more like "AlphaGo is different from humans, airplanes are different from birds, and deep translation is different(better?) from human translation."
I think Shallit is largely wrong. The Hof didn’t specifically cite syntax vs semantics but I think that’s what he’s saying and that’s what GT is doing.
>> If mediocre translations can be done now without the requirements Hofstadter imposes, there is just no good reason to expect that excellent translations can't be eventually be achieved without them, at least in the same degree that Hofstadter claims.
Of course, there's no good reason to expect that excellent translations _can_ eventually be achieved in this manner, either. We just have to wait and see, according to Jeffrey Shallit.
On the other hand, we can already see that GT is not just "mediocre"; it fails in specific ways that suggest fundamental weaknesses of the way it does translation. By analogy, it's like having a flying machine that can only fly as long as it's anchored to the ground. There's no good reason to expect that such a machine will ever be able to do anything else than fly around the same spot.
So Hofstatder's article is not a discussion of requirements for good translation, only. It's also a discussion of limitations of Google Translate, that have to be overcome before it can consistently offer good translations.
> The bailingual engine isn’t reading anything—not in the normal human sense of the verb “to read.” It’s processing text. The symbols it’s processing are disconnected from experiences in the world. It has no memories on which to draw, no imagery, no understanding, no meaning residing behind the words it so rapidly flings around.
I had forgotten how edifying Hofstadter's writing was about topics such as this.
DRH's /Le Ton Beau De Marot: In Praise Of The Music Of Language/ is a remarkable book about translation of literature (all of his books are must reads, btw). Still, it seems to me that shallowness is a feature, not a bug, of automatic translation.
Related heads-up: if you ever want to use or test Google Translate, don't make the mistake of assuming Gmail's translator would return its results. I have no idea what it uses, but it sure isn't the same Google Translate. I've seen it produce output far inferior to that of Google Translate for the exact same text.
Google Translate - and all the other translation engines bar some experimental ones - are a step in the right direction. No, they do not translate like a human translator does as they are better compared to transpilers than translators. Be that as it may the emergence of the likes of Google Translate has made it possible for just about anyone who can read and write to get the gist of what is written in another language without needing to get outside help. While the translation might be rickety it generally is possible to get what was written. The next step will be taken when the technology is ready for it, no sooner. Giving the rate at which machine learning or 'AI' is being pushed it won't be that long before human translators are taken out of the loop for most tasks. They'll still have a job translating literature and some legal texts [1] but most business communications will be translated by machine.
[1] even though legal texts should be a prime candidate for machine translation as they are written in something resembling human byte code to start with.
In 1968 the Sunday Times ran a competition to translate a poem by Baudelaire ("Je suis comme le roi d'un pays pluvieux") into English.
The poet Nicholas Moore read about the competition and, apparently angered by the fruitlessness of the task, entered it 31 separate times with 31 different poems, many submitted pseudonymously and in the styles of other poets.
One of them even begins "I'm like the Winner of the Competition / the one who wrote the strong, rewarding phrase..."
None of his entries won, but his anger ("All I have against translation is that it can't be done!") carried him a surprisingly long way.
My sister is a professional translator and when companies pay her they generally do it because they want really good translations not because they just want people to understand the content. This can be for semi-legalistic reasons (though she's not a legal translator per se, but she has done a fair bit of work related to the EU) or for reasons of professional reputation. Literary translation, although not her mainstay, is also obviously really hard to do well. She's not worried about Google Translate at all yet, although she does have the advantage that one of her languages is rare for native English speakers and she is an extremely good writer in English.
I have a cluster of friends who are professional translators and interpreters and the translators have complained to me a lot about a machine translation program called Trados, that they are pretty much forced to use in their day-to-day jobs.
The complaint I heard was that translation agencies hire translators with contracts that state that 80% of the translation work will be performed by Trados and the human translator will only be paid for 20% of it (I obviously don't know exact numbers!).
Of course, in practice, Trados' translations are nowhere near 80% of the work needed to translate some text, to the point that the human translator ends up doing the bulk of the work and being paid for only a small fraction of it.
It looks like translators are already in trouble with machine translation software, even if they can still do the job way better than that software.
I think we can expect to see this pattern a lot more often in the future: automation taking over jobs not because it's better at them, but because it's more profitable that way.
Trados is not a machine translation tool; it is a CAT (computer-aided translation) tool, which helps human translators do their jobs more efficiently. It does not produce translations.
Trados may additionally offer MT services, and it probably provides easy access to MT services within the tool, but it's not fair to blame Trados itself for poor translations, as the human must choose what ultimately to output for each input string.
> contracts that state that 80% of the translation work will be performed by Trados and the human translator will only be paid for 20% of it
This is probably referring again not to MT, but to "leveraging" existing translations. This is basically a lookup (usually with some fuzziness to allow n% matches where e.g. 80% < n < 100%) against a database of existing translations. If you are translating a document that has been translated before, and only x% of it has changed, then it doesn't make sense for the whole thing to be re-translated.
The effectiveness of Trados or other MT software is entirely dependent on the quality of the glossaries and translation memories.
Both of these are built up over time, the bigger and better (and more specific) they are, the less the translator has to be involved.
If independent, having their own glossaries and TMs boost their efficiency and productivity.
If they are working on the 20%, improving the company TMs is, in the short term, eating their own profit. In the long term, they are training their own replacement.
The author makes one point that is wrong. The current deep learning version of Google Translate specifically didn’t all of the available corpus and data that the old translation uses, simply because the new engine takes much longer to train with the all the data. Google internally looked at statistics and decided that the new engine is aleeady better so there’s no need to feed it more data. I believe they decided that a reasonable time for training (around two weeks IIRC) is more important.
So any want of quality is definitely a pragmatic choice made by Google, but not for want of data.
He never did explain why Frank and his Danish friend used Google Translate despite both being fluent in each other's native languages. I read right through mainly to try and find that out.
On the other hand, it's interesting how well Google Translate translates philosophical text. Just compare the translations below.
Gadamer's text from Truth and Method:
Die folgenden Untersuchungen haben es mit dem hermeneutischen Problem zu tun. Das Phänomen des Verstehens und der rechten Auslegung des Verstandenen ist nicht nur ein Spezialproblem der geisteswissenschaftlichen Methodenlehre. Es hat von alters her auch eine theologische und eine juristische Hermeneutik gegeben, die nicht so sehr wissenschaftstheoretischen Charakters waren, als"vielmehr dem praktischen Verhalten des durch die Wissenschaft ausgebildeten Richters oder Pfarrers entsprachen und ihm dienten.
Weinsheimer and Marshall translation:
These studies are concerned with the problem of hermeneutics. The phenomenon of understanding and of the correct interpretation of what has been understood is not a problem specific to the methodology of the human sciences alone. There has long been a theological and a legal hermeneutics, which were not so much theoretical as corrolary and ancillary to the practical activity of the judge or clergyman who had completed his theoretical trainin
Google translate:
The following investigations have to do with the hermeneutic problem. The phenomenon of understanding and the right interpretation of the understanding is not only a special problem of the humanistic methodology. There has also been a theological and juridical hermeneutics from ancient times, which were not so much scientific-theoretical in character as they corresponded to and served the practical behavior of the scientist or pastor trained by science.
There's no need to wonder, one can be virtually certain that every "classic" work that has been translated to many languages is included in the training data of major translation systems.
MT world runs on data, it includes as much as it can, and a book translation wouldn't be included only if its translations are not readily available in digital form - and pretty much all famous books have been digitized.
It already works very well, especially after its improvement in the recent years after using neural networks. I've frequently been surprised by the quality of the translation it provides to Japanese sentences, not to mention between German and English. The test cases I put in in some other languages (e.g. Chinese) are really promising too.
Machine translation is a really hard problem. The messiness of language as a system, the importance of context in daily conversation etc. all play a part. Another layer of complexity is the gap between everyday usage and official, written form, which is also being tackled by researchers. You have to put this thing into perspective. Many of the old rule-based/Chomskyan software have been simply unusable for decades. New statistical approaches have been in use for barely 10 years, and industrial deep learning less than half a decade. There are still much more to come. The hype IMO is well justified.
These NNs have fewer neurons than an insect brain. They have to use so little computing power they can be provided for free to everyone in the world. Of course they have limitations. But there's been exponential progress in the last few years, and it's amazing they can do as much as they do.
I've found, for Korean, at least, Naver Papago is miles ahead of Google Translate - Especially while decoding some unnatural phrases.
'백조가 연못에 있지만 나는 안봐' has its [implicit] object pronoun dropped for its second verb when translated through Google, but not when translated through Papago.
I use Google Translate extensively to help write things in Japanese - considered (one of) the most difficult languages in the world.
I show the results to Japanese people who nearly always tell me it looks just fine to them.
I have also passed off translations as my own work on tutoring sites like Lang8 and had natives correct "my work". They often will give a slightly different wording, but I have never had an "WTF does this mean?" type response.
----
I think it is important to distinguish the difference between a perfect translation with nuance, and a simple "I need to make this point" translation that is what we need most of the time.
> [...] Japanese - considered (one of) the most difficult languages in the world.
Just in case, one of the most difficult languages to learn by native English speakers. It is pretty easy for Koreans for example---they share the same sentence order (subject-object-verb) and many words frequently share the etymology due to throughout Chinese influence.
More quantitively, Idibon had once catalogued [1] the weirdness of natural languages compared to others. English ranked 33rd place out of 239 languages. Quoting the original, "[p]art of this is to say that some of the languages you take for granted as being normal (like English, Spanish, or German) consistently do things differently than most of the other languages in the world".
>I think it is important to distinguish the difference between a perfect translation with nuance, and a simple "I need to make this point" translation that is what we need most of the time.
Isn't this exactly the point though? A different language in a different culture would convey an idea in a different way.
I'd love to know what exactly you're translating. I have experience with translations from work. I've never met a native speaker say that a translation was any good. Usually, "I understand the meaning but you would never say it that way."
It might be worth looking into the pidgin languages that tend to emerge when two cultures meet and start to trade without the intervention of a officially appointed/educated class of arbitrators i.e. machine translated text might be seen as a kind of pidgin language thats not really any of the two languages in question.
>> “South study walking” is not an official position, before the Qing era this is just a “messenger,” generally by the then imperial intellectuals Hanlin to serve as. South study in the Hanlin officials in the “select chencai only goods and excellent” into the value, called “South study walking.” Because of the close to the emperor, the emperor’s decision to have a certain influence. Yongzheng later set up “military aircraft,” the Minister of the military machine, full-time, although the study is still Hanlin into the value, but has no participation in government affairs. Scholars in the Qing Dynasty into the value of the South study proud. Many scholars and scholars in the early Qing Dynasty into the south through the study.
> Is this actually in English? Of course we all agree that it’s made of English words (for the most part, anyway), but does that imply that it’s a passage in English? To my mind, since the above paragraph contains no meaning, it’s not in English; it’s just a jumble made of English ingredients—a random word salad, an incoherent hodgepodge.
> In case you’re curious, here’s my version of the same passage (it took me hours)
I stopped reading here for now, to avoid having his translation affect what I am about to do.
What Hofstadter doesn't really go into is that I can still manage to extract some information from the machine translation, compared to none for the original Chinese. Not only that, interpreting machine translations itself is a skill. In a sense, instead of learning a second language, one learns to translate poorly machine translated English. Of course, one can still ask whether that's a good thing or not. Here's my attempt:
> “South study walking” is not an official position, before the Qing era this is just a “messenger,” generally by the then imperial intellectuals Hanlin to serve as.
“South study walking” is GT's best attempt at labelling an unofficial position taken by intellectuals, comparable to being a messenger for the emperor.
> South study in the Hanlin officials in the “select chencai only goods and excellent” into the value, called “South study walking.”
It was a position only available to highly-qualified <Hanlin officials>. Quick google search for "Hanlin": The Hanlin Academy (Chinese: 翰林院; pinyin: Hànlín Yuàn; literally: "Brush Wood Court"; Manchu: bithei yamun) was an academic and administrative institution founded in the eighth-century Tang China by Emperor Xuanzong in Chang'an.https://en.wikipedia.org/wiki/Hanlin_Academy
.. so people from Hanlin academy, suggesting the position was administrative in nature.
> Because of the close to the emperor, the emperor’s decision to have a certain influence.
The position was close to the emperor, giving those who held it some influence over him.
> Yongzheng later set up “military aircraft,” the Minister of the military machine, full-time, although the study is still Hanlin into the value, but has no participation in government affairs.
"Study is still Hanlin" is likely referring to the “South study walking” position, since we established the connection to Hanlin earlier. With that, this reads as: Yongzheng set up a ministry of defence, which meant the position was excluded from direct government affairs, although there was still value in having the position.
> Scholars in the Qing Dynasty into the value of the South study proud. Many scholars and scholars in the early Qing Dynasty into the south through the study.
Many scholars in the Qing Dynasty have taken the position of "south study walking", and it was a prestigious position.
I'm sure this is terrible, full of errors, and even the information I correctly inferred undoubtedly misses a lot of nuance, but again: it gives some sense of the information the original passage contains.
So here is Hofstadter's translation.
>> The nan-shufang-xingzou (“South Study special aide”) was not an official position, but in the early Qing Dynasty it was a special role generally filled by whoever was the emperor’s current intellectual academician. The group of academicians who worked in the imperial palace’s south study would choose, among themselves, someone of great talent and good character to serve as ghostwriter for the emperor, and always to be at the emperor’s beck and call; that is why this role was called “South Study special aide.” The South Study aide, being so close to the emperor, was clearly in a position to influence the latter’s policy decisions. However, after Emperor Yongzheng established an official military ministry with a minister and various lower positions, the South Study aide, despite still being in the service of the emperor, no longer played a major role in governmental decision-making. Nonetheless, Qing Dynasty scholars were eager for the glory of working in the emperor’s south study, and during the early part of that dynasty, quite a few famous scholars served the emperor as South Study special aides.
Well, that definitely reads a lot better, but I wasn't that far off in terms of meaning of the text. And it didn't take me hours (writing this comment took a long time though).
I absolutely agree that human translation by experts is an art, that it produces much, much better results, and that we should not let it be devalued. But the value in getting a quick impression, even if flawed, through machine translation should not be undervalued either. It has a very different application. On social platforms, for example, it is the difference between being completely out of the loop of a conversation or still somewhat following it and being able to ask a question for clarification in a shared language.
I don't know what translation engine Bing is using, but I find that combining Google Translate and Bing results gets me pretty close to the real thing.
I wish teams making those two could work together :(
The 锺书 example matches my experiences with Chinese through Google Translate.
Chinese is a language where you'll often redundantly say things because each character may have (a) multiple meanings and (b) a lot of homophones, even accounting for tone.
So the word for vegetable(s) is "蔬菜", which really is two different words for "vegetable" put together.
I suspect that the engine has "learned" that words can be thrown away sometimes.
I bet what happened with 锺书 was that Translate decided to throw away 锺 in one of the cases because it didn't know what to do with it, leaving it with 书 ("book").
-------------
Regarding the anecdote about the Danish-speaking friend, this isn't too uncommon. For example, I natively speak Marathi but spoken/written Marathi differ a lot and I don't have much practice reading/writing, so I make spelling/grammar mistakes. So using Google Translate as a crutch (and then modifying/verifying the output) is great. Though I've found out that it's not really good at non-EU languages (The EU translates all of its documents into the languages of its members so this forms an excellent corpus of professional translations for Google Translate) and that my written Marathi is often better.
-------------
My favorite example of Google Translate limitations is what happens if you ask it to translate "Yes." and "No." to Chinese. You get the answer 是 and 没有, which literally translate to "am" and "don't have". (there are no conjugations and kind of no tenses, so by "am" I mean "whatever conjugation of the English word 'to be' makes sense in this context")
The thing is, Chinese doesn't really have words for Yes and No. 是的/不是 sometimes works (I think this is more because there are implicit 是s in verbless sentences). But the basic idea is that if you want to answer a question, you repeat the verb. "Do you eat meat?" "Don't eat." or "Do you read books?" "Read.". 是 and 有 are pretty common (as are "to be" and "to have" in English), so it seems like it picked up on the most common yes/no answers. For some reason, it picked a different one for no than it did for yes.
Now, this isn't exactly an example of the machine translation sucking, but more of the translation not having an outlet to express ambiguity at all. The article gives examples involving this as well, but the basic idea is that some languages do not allow for the same kind of ambiguity that others do. You should be able to display something about that when translating from an ambiguous term to a specific one.
Yes/No is an example of a set of ambiguous terms in English where Chinese has no similar ambiguous terms. In the other direction, as I already explained, 是 (or any verb!) in Chinese is an example of an ambiguous term that doesn't translate to English because the tense need not be fully specified (Chinese does tense from context, and even then it may not be as fully specified as it is in english), and English requires you to handle tense.
1 day ago, 3 days ago... there is a serious problem with the short attention span of HN. No sensible forum should split threads like that. Some interesting discussions take longer than a day to execute, and there is no reason why a thread could not be alive for years, but HN actively discourages it - there is no interface for discovery besides the novelty firehose of frontpage+new, and voting is forbidden after a certain age (possibly replying? I've never tried).
As for attention span, threads often go for longer than a day but rarely go for longer than a few days. That's just how the community behaves and if, as I assume, it's true of internet forums generally, there's little to be done about it.
I see this as an inevitable consequence of several deliberate design features. It's a link-sharing site, and people get karma for popular links. The votes serve to make the items rise on the front page, getting more votes, hence more karma.
So people will rush to be first, and there is no serious mechanism to check whether the same story, or even the nearly exact same link has been submitted before.
But anything more sophisticated will be more complex, harder to understand, more fragile, and possibly less popular. It's not hard to see several things that might make it better, but it's uncertain, more work, and guaranteed to make the system less transparent.
<fx: shrug />
To some extent it is what it is. But FWIW, I agree.
Reddit does a simple link comparison to see if the article has been submitted before, which does limit the double-posts, however some people then intentionally subvert the process by running the link through a processor like outline.com or archive.is, or simply finding alternate articles from other sources.
This is unfortunate as the thing I really want to see is an in-depth conversation about a topic more than an individual article.
An ironic joke that you don’t find funny doesn’t make it nonsense. I was aware he was not a bot and was simply commenting with tongue firmly planted in cheek that he seems to expend bot like effort on posts exactly like these.
Clearly, getting machine-like reading comprehension is easy: just react instinctively to the lede, utterly ignore the core of the article as TL;DR, put words in author's mouth for good measure. That is not quite the kind of equivalence that the field is striving for: "humans can't read, machines can't either, QED."
Your response simplifies a pretty specialized occupation. Translating a paragraph is not simple math. Most of the time ‘understanding’ natural language is AI complete. Translation is not just looking up words but also considering culture and history. So maybe don’t be so sure that with the correct ‘magical’ technique we can achieve human level translation...
Do you speak a second human language that’s not closely related to your first?
English and Russian, say?
Because I find most data scientists that do tend to think language translation may be one of the last things we solve. I think it requires hard AI, honestly.
Maybe look at the progress we have made in last twenty years? Yes, there are examples of machine translation failing miserably, but it has also reached a level which allow people to actually use it for communication (as evidenced in real life and mentioned in the article). I think that's already quite an achievement.
We should not be too harsh on computers. Humans are not perfect either. Instead of judging machine translation based on few random cases, we should be evaluating against number of human translators of various experience levels.
The level of communication it enables is thin. Far from even simple conversations and relies heavily on humans having a “theory of non-mind” of the machine.
I speak three: Russian (native), English (mistaken for native speaker by native speakers) and Hebrew (average). I may not be an expert in AI though (couple of courses, hobby projects and worked alongside AI researchers on the non-ML side of some apps). Still, looking at the methods we already have and recent successes (like Yandex's linguistic work), it doesn't like as hard a problem as hard AI with human-like cognition.
What's going to happen is that we humans are going to become more machine-like, as we loose our appreciation of our own sophistication & subtlety, so that most people will be satisfied with machine-translation, writing, art and music never having known that there are higher levels
The word is "χελιδόνι", meaning "swallow" (the bird) in Greek. For as long as I've been trying this tiny little experiment, Google has been translating it to the French word "avaler"- the verb "to swallow".
https://translate.google.com/#el/fr/%CF%87%CE%B5%CE%BB%CE%B9...
Once in a while, a different translation appears- "machaon", which is a kind of butterfly, the Old World swallow-tail. This is slightly closer but still absurd. If GT can make the connection to "swallow-tail", how can it not see the connection to the bird, "swallow"?
The problem seems to be that, in order to translate from Greek to French, Google goes via English. A great big chunk of context is lost in the process, especially since now the translation between two languages that have different forms for male and female nouns goes through a third language that does not.
So for example, I've seen the same gender-reversal as Hofstadter reports in his article. The following string in Greek means "I saw my teacher and she said hello to me".
The following are the French and English translations by GT: This seems to happen because Google's English language model has learned that "he" is found in the context of "teacher" and "said" more often than "she" is. But of course, such a statistical association is, well, meaningless and for that useless when it comes to translation, where you actually need to know when the least likely case is correct.Generally, it looks like the complete abandonment of any attempt at representing meaning, and relying instead on text statistics to do meaning-intensive work like translation, is producing a lot of nonsense. And that's not a criticism of Google Translate only. I think professor Chomsky might "win" that old debate with prof. Norvig, after all.