> The particular paragraph is mangled to the point of being gibberish

You just discovered the poor quality of LSTM generated text. It might come as a surprise to you, but it's legend between AI researchers :-(

If it's not translation (such as in chatbots, image captioning, or computer generated text in general), it's even worse. Unfortunately we can't generate text that makes sense. We can only do it in very specific cases and with lots of errors. Other language models are even worse than LSTMs.

The problem is that automated translators don't have real world experience and can't think causally. Humans can, because we are in possession of a body and live inside a complex world. We can experiment to test our hypothesis, GT only sees text and can't test anything.

> The particular paragraph is mangled to the point of being gibberish

I actually disagree it's gibberish. I could understand enough to make sense of the situation (the part about blades in his leg puzzled me, but I understood about Maria being an ex girlfriend even if the sentence didn't resemble actual human language). This translation is by definition not gibberish.

> But if we hung out with Vasiliki, Mitsos and Maria last year and you put that on Facebook, I'd at least know enough to send my sympathy to Mitsos and have a good chuckle over Vasiliki's romantic endeavours, which would probably be what you intended - - which is pretty much the definition of "good enough".

Exactly!

>> This translation is by definition not gibberish.

You have a definition of gibberish? ~.^

Yes: "meaningless or unintelligible talk or writing." (definition #1 from http://www.dictionary.com/browse/gibberish).

If I can extract a reasonably accurate understanding from a paragraph, then by definition it's not meaningless nor unintelligible.

Btw, the definition above is missing an important point: that all language is, in a sense, gibberish.

What I mean by that is that all the language structures we choose to convey a certain meaning are arbitrary and, by themselves, completely meaningless. We assign meaning to them by convention (and the fact that this convention keeps changing is why we have different languages and why even the "same" language sounds very different a few years down the line). It is when this convention is breached that our ability to understand the intended meaning is compromised.

I'd liken this to adding noise to a signal. The more a passage of text deviates from the conventional structures for the intended meaning, the less the information that can be safely extracted from it. The more the noise, the less likely that 100% of people hearing or reading the utterance will understand 100% of its indended meaning.

So in other words- it's still gibberirsh even if you can understand it. Because maybe you personally can understand it, but any number of other people will not- people with different language abilities, that is.

To draw an analogy- maybe you personally don't need a vaccine for chicken pox because you are immune to it. That doesn't make the vaccine unnecessary. Others don't have your immunity. etc.

> that all language is, in a sense, gibberish.

No, I disagree with this premise. It is contrary to the definition of gibberish I'm using (and which I linked to). Since we disagree on this fundamental issue, the rest of this conversation is very difficult.

If all language is gibberish, like you say, then it's ok for Google Translate to produce gibberish. (Sorry, that doesn't make sense to me!)

I think what I say is not controversial. Maybe my turn of phrase is not very common. What I say is that language structures don't have inherent meaning. We associate them with meaning in a one-to-one relation, but the structure side of the relation can be anything. This is why it's possible to say the same thing, i.e. convey the same meaning, in many different languages, i.e. relating it to many different structures.

So language utterances mean nothing by themselves. They only have the meaning we assign to them by convention. It is in this sense that I say that those utterances are gibberish.

This might sound weird to you, but if you think about it, most natural language processing tasks basically come down to performing a search for the meaning that is related to a given structure. What makes this search very difficult is that structures are inherently meaningless, so there's no logical reason why a given string might mean something particular. Essentially, we try to teach the NLP system to learn our conventions. But that is very hard to do, because we don't exactly understand how these conventions work, ourselves.

So it's not OK for GT to produce random gibberish- because there is only one set of strings of gibberish that is associated with the intended meaning of a sentence in an original language.

And, I guess that's my point. If GT's search fails to find the right string then it's not doing very well.

It's not controversial to claim everything in the world (physical or imaginary) is essentially meaningless except for the meaning we humans assign it. Then there's the consensus of meaning, etc, etc. Welcome to philosophy! :)

> So it's not OK for GT to produce random gibberish

Thankfully, it doesn't in most cases. If it were random, we wouldn't be able to understand it (please, don't ask anything that starts with "how do you know...?". I know. Let's drop that angle, please!) The chance of all of us understanding roughly the same idea from random gibberish is effectively zero. Hence, it's not random gibberish.

But we don't all understand the same thing from language that doesn't closely follow our agreed rules of language structure. Specifically in our example, not everyone will understand what the mangled translation of my text says.

So this is something to wonder about: why is it that we all speak in a certain way? If we can understand language with unfamiliar structure, why does language follow set patterns? Why don't we all speak any old way we like?

Mind you, I'm not saying I have any answer to this. Because it's actually a major question. You could, for example, observe that people will often break from accepted conventions when speaking or writing and it's hard to even measure the deviation from accepted conventions "in the wild" (as opposed to more formal text or speech). And yet there does seem to be a set of patterns of utterances that we observe when speaking or writing in our respective languages. There are rules, in language, and these rules are observed. Why is that?

Understanding the answer to this question -what makes language, language, rather than random noise- could tremendously boost NLP tasks of all sorts. To be honest, I don't see anyone asking it though. Instead people have become comfortable with the paradox of the random noise (sorry, I insist) generated by computer programs, and treat it as an engineering problem to be solved, rather than an opportunity to understand human language by looking at its difference to machine language (if I may).

>> (please, don't ask anything that starts with "how do you know...?". I know. Let's drop that angle, please!)

I understand if you don't want to spend time discussing this, but I hope you understand that, unless you justify your thinking, you can't expect anyone else to agree with it.

In any case, "how do you know" is the fundamental question you have to ask if you want to advance your knowledge. That's how it all begins- by challengine assumptions (yours or others').

> Specifically in our example, not everyone will understand what the mangled translation of my text says.

Not everyone will understand your hand-made translation, either. Human understanding works by consensus. Someone will be left out of this consensus almost always. It doesn't mean the automated translation is gibberish. If enough people understand the automated translation, then by definition it's not gibberish, because random noise doesn't carry enough information for you to get the idea about Maria, Vassiliki, the hospital or the broken leg.

> I understand if you don't want to spend time discussing this, but I hope you understand that, unless you justify your thinking, you can't expect anyone else to agree with it.

I have justified it. I asked you to drop it because we're running in circles, not because I don't have arguments. Your argument seems solipsistic to me: "but how do you know?" -- I know because it's my experience and because other people agree with me. How do you know your translation is good, anyway? Because it's what you think and because other people agreed with you. Because you managed to successfully -- as assessed by yourself -- communicate the idea you wanted.

The difference is that I can point at the ways that GT fails, in terms of specific grammatical and semantic errors, whereas you can only report some examples of people who agree with you. Not trying to be offensive, but your definition of gibberish is subjective whereas mine is objective and quantifiable (in one word: ungrammaticality).

To go back to an earlier example of learning to solve mathematical problems with machine learning- let's call it machine arithmetic. Say we developed a machine arithmetic system that consistently estimated the sum of 4 + 4 to be 7. That's consistently close to the correct answer, but it's also consistently not the correct answer.

So, to pose yet another question: a system trained to solve maths problems who got it almost right 100% of the time, would be considered pretty rubbish. Why is GT not? It's computing language, not maths, but its computation is still very often completely wrong in specific, measurable ways that really leave no room for subjective interpretation.

> The difference is that I can point at the ways that GT fails, in terms of specific grammatical and semantic errors, whereas you can only report some examples of people who agree with you

What matters for the purpose of human communication is the latter, not the former. Success at communication trumps your grammatical errors (where beauty is not concerned, of course. If you're going to argue automated translation produces ugly results, then you'll hear no argument from me!).

> Not trying to be offensive, but your definition of gibberish is subjective

You're not offensive; you're arbitrary. I provided you with a dictionary definition of gibberish, while your definition seems to be suspiciously ad hoc. You've essentially defined gibberish as "whatever output automated translation currently produces", which makes this debate pointless.

Math and natural language are different. Again, the standards are different, just as in your example with child's language. No, we do not judge a computer's ability to "translate" the same as we judge a young kid's ability to speak. There's nothing surprising about these two standards being different.

Your example with Math is not particularly relevant. However, since your background seems not to be computer science [1], maybe you're unfamiliar with heuristics: in the context of computer science, a heuristic is a technique where you take an optimization problem (e.g. "finding the minimum of something", "the shortest route to somewhere", etc) which is computationally untractable and provide a "good enough" answer which can be computed. The definition of what is "good enough" is, of course, relevant to the application of a particular heuristic in a specific context. In this sense, for some heuristic and its application, maybe 7 is a "good enough" answer even though the right answer is 8.

[1] This was written before I read in another reply of yours that you claim to be working on a PhD in Machine Learning. Fair enough! But even if you're familiar with heuristics, I think my point stands.

>> What matters for the purpose of human communication is the latter, not the former. Success at communication trumps your grammatical errors (...)

But grammatical errors cause communication to fail. Because they introduce (measurable) noise to the signal, which obviously makes it harder to extract information from it.

Btw, grammaticality is another uncontroversial measure of the performance of NLP systems that are required to generate language (like machine translation does). There is some theory behind the idea that well-formed sentences are inherently more intelligible, that will take a lot more to overturn than "it works for me".

To draw yet another analogy- homeopathy "works" for some people: they feel better when they take its concoctions. That doesn't make homeopathy objectively useful.

>> in the context of computer science, a heuristic is a technique where you take an optimization problem (e.g. "finding the minimum of something", "the shortest route to somewhere", etc) which is computationally untractable and provide a "good enough" answer which can be computed.

Yes, looking at machine translation as a heuristic is a good point. Like I say in other comments, the problem of machine translation has no good definition. That's because it's very difficult to define what is a "good translation" in a formal and principled manner. Hence, we're left with heuristics like "how close this automatic translation is to this human translation" (where the human translation is whatever happens to be available, rather than something chosen for its quality or other attributes).

This is actually exactly what machine translation systems do. They are made to optimise a measure of error over examples of human translation. So GT is actually designed to do what you say is not important, produce grammatically and syntactically correct language, like its examples are.

In that, it fails. It may achieve your goal of "communication" but that is not really a heuristic, it's more of a quality that you, personally (and obviously other people too) assign some value to. But it's far from a universally accepted measure of machine translation quality. You might say that "communication" is a business goal whereas matching human translations is an engineering goal.

As a P.S., I'm not "claiming" to be working on a PhD. The information is in my profile with links to my linkedin and github :)

> But grammatical errors cause communication to fail. Because they introduce (measurable) noise to the signal, which obviously makes it harder to extract information from it.

But when communication doesn't fail, grammatical errors don't matter that much. Because communication success is its own evidence (cue your "but how do you know [...]?" -- please don't. We've been down that road and your argument is unconvincing and solipsistic).

You say grammar correctness is another uncontroversial measure of NLP, but nobody was arguing against this. Of course it is. You're shifting the goalpost. Grammar matters. You can tell when something is ungrammatical but whether this affects communication is another issue. It's also a matter of degree: something can be so ungrammatical it conveys no meaning; and something can be ungrammatical enough to sound "bad", but still convey its intended meaning.

> Hence, we're left with heuristics like "how close this automatic translation is to this human translation"

That's one heuristic, sure, and a pretty good one! Not the only one, though. In any case, you missed the point that your Math example was not very good, because it ignored that in a heuristic, what you proposed as a bad "translation" could actually be a good one. In the realm of human language translations, it's never a case of 4+4=8.

> As a P.S., I'm not "claiming" to be working on a PhD. The information is in my profile with links to my linkedin and github :)

My mistake then! No offense intended.

PS: one more thing: I've looked at your comments history and I actually agree with your skepticism about AI. I think it's overhyped, especially by people with no background and no formal understanding about it -- like Eliezer, who you rightfully mock in your profile, and many other Prophets of the AI Apocalypse (or AI Utopia, depending on their mindset). So consider me a skeptic as well. We're more in agreement that it seems! I just don't think the kind of translation we're talking about is anywhere close to General AI or "understanding" of any form, and I can get excited about a "limited breakthrough" even though it's not actually about AI :)

Well, alright then. Let's agree to disagree so I won't say anything else. This thread's text is already pushing very aggressively towards the right side of my browser window anyway.

>> No offense intended.

... and no offense taken, absolutely.

We'll probably get the chance to argue again in the future, anyway, given that you're interested in the same subjects as I am :)

In that case, if that sentence above is not gibberish, why doesn't everyone speak that way?

Because non-gibberish doesn't mean "the best/most common way to communicate". If you speak in English like Yoda, the character from Star Wars, you're perfectly intelligible -- hence, you're not speaking gibberish -- even though (almost) no-one speaks that way in real life.

Gibberish means nonsense. If most people understand you, it's not nonsense.

But how do you know most people understand you if you're not speaking like most people speak?

You're right: I do not know for sure that most people understand Google Translate's translations (with more or less effort). I assume they do because I do, and because other people I know also do.

>> I have had fully meaningful (if not exactly deeply philosophical) conversations

Any conversation can easily become "philosophical" when Google Translate messes up and inserts random garbage instead of intended meaning, and people, instead of dismissing translation as garbage, decide to seek some deeper meaning in it :)