10

My question relates to full-text translators that are not specifically based on LLMs. My current understanding is that the term Generative AI goes beyond LLMs and that the full-text translators (especially those which are based on artificial neural networks) also fall into this category.

In the Wikipedia article about Generative AI, I could only find the statement that generative AI systems such as ChatGPT are also used for translations. That is quite obvious, but this does not answer the question of whether other full text translators are also commonly referred to as Generative AI.

IBM research defines Generative AI as

Generative AI refers to deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on.

AFAIK DeepL, Google Translate, Bing Translate but also a few not so well-known systems falls clearly into that definition.

My question was triggered by an answer on Meta Stack Overflow that contained the following sentence:

Besides, the banner clearly states: "Answers generated by artificial intelligence tools". Translations aren't generated answers. They're translations.

with the implicit conclusion that it is clear to everyone that pure translators do not count as generative AI, hence it does not need any further explanation. Other participants in that thread seem to agree to that point of view. However, I think their use of terminology is not the typical use in the field of AI, and I would like to hear what the experts say.

Doc Brown
  • 202
  • 2
  • 10

5 Answers5

8

Generative AI, as defined by IBM research, refers to deep-learning models capable of creating new content, be it text, images, or other media, based on their training data. This definition indeed encompasses models like GPT-3 or GPT-4, which can generate text in various styles and formats, including translations.

However, when it comes to full-text translators like Google Translate, DeepL, or Bing Translate, there's a nuanced difference. These systems are typically based on neural machine translation (NMT) models, a specific application of deep learning tailored for the task of translating text from one language to another. While these NMT systems are indeed 'generative' in the sense that they produce new text in a target language, their primary function is not to create original content but to convert existing content from one language to another as accurately as possible.

FledDev
  • 204
  • 2
  • 5
    I would also say that typical models that people call "Generative AI" like ChatGPT and Dall-E, are not meant to create original content either, so that would not be a good criteria to define generative models. – Dr. Snoopy Jan 25 '24 at 13:33
  • 5
    Wait... What are they meant to do then, if not "create original content"? – Cerbrus Jan 25 '24 at 13:34
  • So is my understanding correct that in the field of AI, "Generative AI" is a term which can include more than just LLMs? Or would you say that this interpretation is quite unusual and too far-fetched? – Doc Brown Jan 25 '24 at 13:36
  • 4
    Depending on the model, they are trained to predict next word given some context (LLMs), or to imitate a training set of natural images (Stable Difussion and Dall-E) , creating new content is a strong claim that is generally unproven. For example see the NYT lawsuit against OpenAI. – Dr. Snoopy Jan 25 '24 at 16:03
  • 2
    @DocBrown Correct, Generative AI existed long before LLMs. – Dr. Snoopy Jan 25 '24 at 16:03
  • 1
    "see the NYT lawsuit against OpenAI." is not very helpful. If you ask a LLM like ChatGPT to generate a story about imaginary characters you made up living on the moon, it will. You can't tell me that that isn't "new content"... – Cerbrus Jan 25 '24 at 16:15
  • 3
    @Cerbrus It will plagiarize an existing story with your characters, this has been shown over and over, the NYT lawsuit just shows that it can produce word by word content from the NYT. It applies to almost everything that is generated, another example about jokes: https://arstechnica.com/information-technology/2023/06/researchers-discover-that-chatgpt-prefers-repeating-25-jokes-over-and-over/ – Dr. Snoopy Jan 25 '24 at 19:18
  • 2
    @Dr.Snoopy "It will" is not the same as "it can". The lawsuit claims to have shown you can get it to reproduce specific content. To show that this is the only thing it does, you have to show all content it produces is plagiarism, which specific examples fail to prove. – Yakk Jan 25 '24 at 20:13
  • @Yakk No, it is a machine learning model, it only works on its training set boundaries, to claim novelty in generating content, you need specific evidence that has not been produced at all, while there is plenty of research showing the contrary, there is even a recent paper proving that Transformers do not generalize outside of their training set: https://arxiv.org/abs/2311.00871 – Dr. Snoopy Jan 25 '24 at 20:23
  • 1
    @Dr.Snoopy If I remember correctly, ChatGPT had also closely replicated a news article written after its cutoff, so it couldn't have been plagiarizing – Esther Jan 25 '24 at 21:30
  • @Esther may be the news article itself was plagiarized from something that existed before chatgpt's cutoff – user17915 Jan 26 '24 at 07:08
  • 1
    @Dr.Snoopy I mean, a pseudo-RNG produces novel output in terms of pixel static. Such a step is part of art generation by techniques like stable diffusion. It might not be interesting novel content, but one of the algorithms inputs are samples from a larger space of novel "noise" than the total number of images ever uploaded to the internet. Unless there is convergence in the output image from the random source, it cannot only be duplicating images, there aren't enough holes to put the pigeons in. If you want interestingly novel content, no guarantee, but it is novel. – Yakk Jan 26 '24 at 14:55
  • @Yakk Sorry but your explanation did not make any sense to me, Diffusion models just estimate probability distributions, there is no magic there. – Dr. Snoopy Jan 28 '24 at 21:27
7

The distinction between "generative" and "non-generative" AI isn't an especially useful one. A language model (to first approximation) takes a sentence as input and tells you how probable it is that a native speaker would phrase it that way. Any remotely decent translation system is going to contain a language model, either as a separate component or integrated into the translation model proper. This helps ensure that the text it outputs is grammatically correct and scans well. Without a language model, you get a word-for-word translation that won't sound natural at all.

But if you have a probability distribution over the next word given the previous words, not only can you answer how probable an existing sentence is, but you can take a series of words and sample the next word in proportion to its probability under that model. You then put that word in the context and generate a new one, and so on. Any model that can evaluate the probability of a sentence can be trivially modified to generate a sentence. Similarly, you can take any generative model and output the distribution over the next word instead of just selecting a single one; the product of these successive distributions will be the joint distribution over sentences of that length. The models are exactly the same in either case; only the user interface differs.

(Of course, when setting policies, those user interface details might actually matter, since we might want to encourage users to fix their grammar but discourage them from posting answers when they have no idea if they're correct. But those details are entirely uninteresting mathematically.)

Historically, the use of language models in things like machine translation and voice recognition are the serious uses, and using them to generate text is a neat side task that can be used to sanity check the sort of sentences it thinks are representative of the model. (Text generators can also be fun toys, of course, which is how they're being used in things like ChatGPT.)

Ray
  • 346
  • 4
  • 12
  • Maybe the distinction is not clear cut, but obviously there are AI systems which clearly do not count as GenAI, and others which clearly count as such for the majority of people. In the middle, there is a grey area. Do you think systems like Google Translate belong more clearly into the "non-GenAI" catgegory, the "GenAI" category or somewhere into that grey area? – Doc Brown Jan 25 '24 at 21:19
  • 1
    @DocBrown The problem is that the classification "Generative AI" is not based on strict mathematical or engineering properties. Likewise, banning use of "generative AI" cannot be relaibly based on a model containing some mathematical or enginering element. Translation systems are, to me, generative systems. They sample from a probability distribution of outputs, and to create a translated sequence, usually reinsert what they just sampled as "written so far", just like LLMs performing text completion. The models are very similar. What is different are the task-based constraints on ground truth. – Neil Slater Jan 25 '24 at 22:34
  • @NeilSlater: yes, thanks. This confirms to me that Generative AI is a vague term which is commonly used for a lot more systems than the current AI ban on Stack Overflow seems to be intended for. Hence I think it needs to be clarified whenever it is used in a certain context. I miss exactly that clarification in SO's current AI policy text. – Doc Brown Jan 25 '24 at 22:49
  • 4
    @DocBrown I think focusing on a strict definition for the ban is looking at the wrong thing. What the authors of the ban want to prevent is use of low-effort machine assisted answers that have a high chance of looking authoratitve, but that contain mistakes. The ban is a rough heuristic towards that goal. Looking for loopholes or theoretical overreach, either as a poster or as a mod, is missing the point. – Neil Slater Jan 26 '24 at 07:22
  • @NeilSlater: yes, absolutely, but I am not looking for a strict definition for the ban, I am looking for a better heuristics which is not so single-edged as the current phrasing is. My post here, however, is an attempt to question my POV that a better wording is needed, since the term GenAI has IMHO a broader meaning than some of the MSO protagonists are pretending. – Doc Brown Jan 26 '24 at 07:40
  • 1
    @DocBrown It's not that the meaning is broad so much as that the meaning is based on "how it's used" rather than "what it is". Given that, it would probably make sense to phrase the policies as something like "Don't use language models to create semantically significant parts of your answer (as opposed to modifying an answer you wrote, without changing the semantics (e.g., via translation or grammar improvement)." Say what you don't want the users doing instead of trying to exhaustively list all the tools they might use to do it. – Ray Jan 26 '24 at 16:12
  • @Ray: That sounds really good - that is something I would greatly support. I understand that the policy lists examples of well-known tools which are banned, to make it unmistakable clear clear whats forbidden. An additional paragraph about responsible use of AI could state exactly what you wrote - other AI tools might be used for improvements of an answer as long they don't add or change the semantics of any input a user of such a tool provides. – Doc Brown Jan 26 '24 at 17:34
5

Actually it depends, generative models are specific kind of machine learning models. Generative often means a model that models the probability distribution of data $p(x)$, you cannot do translation using this kind of model.

But also you can build conditional generative models, where the probability of data points $x$ conditioned to some input $y$ is modeled $p(x \, | \, y)$, and in this case it is possible to perform a translation task using a generative model, where $x$ is the translated text, and $y$ is the original text.

So it all depends on the model you use for the task, if you use a (conditional ) generative model to perform translation, assuming that the model is trained for that task (like ChatGPT), then yes, translation can be done using generative models.

The concept of translation in AI/ML is more general, to just produce an output given an input, for example, CycleGAN is a generative model that translates between two sets of unpaired images, from horse to zebra and viceversa. And GANs are generative models.

About the specific question on DeepL and Google Translate, it depends on what underlying model they use, if at some point they use generative models to perform translation, then they should also be flagged as generative AI.

In general the concept of Generative AI has... deformed over time... (due to hype). Generative AI has existed long before LLMs and Difussion models like Dall-E and Stable Difussion, just look at Generative Adversarial Networks (GANs),

Dr. Snoopy
  • 1,355
  • 7
  • 12
  • 2
    We're not asking if you can. We're asking if these translation tools are generative. Your answer is basically "we don't know"... Considering the accuracy of the translations, and the tendency for generative AI to generate inconsistent output, I'm fairly sure generative AI doesn't play a role in them. – Cerbrus Jan 25 '24 at 12:11
  • @Cerbrus The OP is asking more generally, if language translation can fall into the category of generative AI, and the answer is definitely Yes. – Dr. Snoopy Jan 25 '24 at 16:05
  • 1
    @Dr.Snoopy: you understood me right. Still I agree with Cerbrus on one point: your answer did not directly answer my question about common usage of the term Generative AI. However, your former comment does, thanks for clarifying. – Doc Brown Jan 25 '24 at 16:46
  • 1
    @DocBrown Good, I integrated that comment into my answer now. – Dr. Snoopy Jan 25 '24 at 19:14
2

The original question has been updated so my answer is being updated to reflect this.

The question being asked can be boiled down to just "What is it that makes something a generative AI?" I am quite familiar in my field of computer vision with Generative Adversarial Networks, but I believe after looking at Large Language Models that they operate on a similar principle. I will go over both of these.

Generative Adversarial Networks

First I will go over the GAN which I know well, though I have not implemented one in almost 8 years. In a GAN, you have 2 machine learning models, one is generative, and one is not. There is a non-generative model called the discriminator who's job is to determine if an image belongs in the dataset or not. Then you have the generative model who is considered an attacker or adversary, who is there to try to fool the discriminator, by producing an image that the discriminator thinks belongs to the dataset, but it does not.

During training we cycle between training the discriminator on the dataset until it cannot be fooled. Then we train the adversary until it can fool the discriminator. Then we add the adversary's images to the dataset and go back to the start again. We continue this process until the discriminator can no longer be trained to tell the difference any more, and the adversary wins. Now the GAN is complete and we discard the discriminator.

The GAN can be altered to have an input parameter where you can select a class, and it will attempt to create an output of that class to fool the discriminator with that class. This is an optional feature, but something many GANs try to implement.

LLM vs GAN

LLMs on the other hand looks to only have one model. If you dig into it though, you will find out there really is 2!.

Creating LLM Vector Stores

The Vector store is a model too! It is a generative model that provides an output to the final model that interprets the output. So you have a model that is trained on a large language database, and a vector store that is trained to output segments of words similar to the input it is given. Both the input you give and the input the vector store outputs is given to the regular model, and it takes that and provides an output to you.

So mechanically, what ties these two types of generative AI together? How can we create a mechanical definition? You have a normal network, either a discriminator, or interpreter, and then you apply to it a generative network, either a adversary or tokenizer(?) respectively during training. When you are done, either you keep them together in the latter case, or you discard the discriminator and publish the adversary in the former case, and you have a generative AI.

For older translation models, they likely did not have the processing power, or technology available to perform these kinds of tasks. This is a relatively new field. They would not have been able to be generative AI. They at most would be comprised of one model during both training and publishing.

Old answer continues below:

> [arXiv:2307.15208][4] states "Generative AI refers to a set of
artificial intelligence techniques and models designed to learn the underlying
patterns and structure of a dataset and generate new data points that plausibly
could be part of the original dataset."

Considering this, what we see is that a generative AI differs from a traditional AI in that it creates new information where none was before. A traditional AI might detect which paintings belong to Raphael, while a generative AI might try to create a new painting in Raphael's style.


I think your misconception here is that all neural network AIs produce information of some kind, but they do not produce new information. The information was already there in the input in some form in a traditional AI.


    

  • The location of the dog was in the picture already, the neural network just pointed to it and said, "Here it is."
    • 

  • The tune matched a Metallica song in a database already, the neural network just correctly identified it.
    • 

  • The information was already encoded in a foreign language, the neural network just decoded the information and then reencoded it into a new language of your choice.
    • 



Now if the model had instead read the foreign language, decoded the information, and used it as a prompt to write a novel in the language of your choice, the contents of which included new information never before seen in that input, now we are talking about generative AI. This is in fact what they do.


A lot of the media talks about how chat GPT can hallucinate it's answers when you ask it straight forward questions, but the truth is, it is generative AI. It is giving you answers that could have conceivably been in the answer set. They just aren't. This is what some experts mean by it hallucinating up new answers. It's pulling them out of the void like a generative AI is supposed to do. It just isn't what the average person expects this type of program to do.


In short, what the difference between a traditional AI and a generative AI is that a traditional AI learns a dataset, and tries to tell you something about the input with respect to the dataset. The generative AI takes the input and tries to produce something similar that it thinks would be from the same grouping in what it believes the dataset should be.
Chthonic One
  • 129
  • 2
  • Strange. It did not appear to be behind a paywall for me, but you are correct in that it is a medical paper, but it is also the correct definition as well. I don't see what the issue is with the definition. https://arxiv.org/pdf/2307.15208.pdf – Chthonic One Jan 25 '24 at 23:01
  • The definition in that paper (about medical imaging) looks like it was written by authors who picked a phrasing which suits their needs in context of that paper. This is nothing I would consider to be a valid statement about the general, common use of the term GenAI in the field of AI, more one possible use of the term in the field. In that context, that is ok and surely correct, but I would not draw any conclusions about the terminolgy for translator systems from it. – Doc Brown Jan 25 '24 at 23:07
  • ... note also I never said I believed all artificial neural network systems count as GenAI, that would have been clearly nonsense. I was talking specifically about translators, where you feed one text in as input, and get a - translated - text as output. My understanding of this process is that is is not uncommon to call this "generation" (of a text in a diffferent language). – Doc Brown Jan 25 '24 at 23:16
  • As I noted in my answer, this is a misconception on your part. Generation in this case has more than one meaning. Yes, it generates text. No it does not generate new information that did not exist in the original message, at least not intentionally. Of course mistranslations do happen, and when they do, new information can be accidentally injected, but that is considered a bug, and not a feature. – Chthonic One Jan 25 '24 at 23:18
  • So you deny that the common usage of the term GenAI in the field can include translator systems (as long as they are based on what you call "traditional AI")? Note I mean the common usage of the term over the last decade and maybe earlier, not the "popular usage of the last year". – Doc Brown Jan 25 '24 at 23:23
  • I deny the fact that the definition used by me or by others here is wrong simply because it is not used in a more general case. The people who create this definition are the ones pioneering the systems, and they do so by writing papers on what they do in specific cases. You can learn what they mean by when they define their terminology in their papers. When all of the papers agree on a definition, you know what it means in the general case. There isn't going to be a paper that defines it for you in the general case. Other than that, the definition is set arbitrarily by those that do this work. – Chthonic One Jan 25 '24 at 23:26
  • 1
    As such, the definition says that a generative AI will produce a result out of nothing that will match what it believes should be in the dataset. A translation AI does not produce a result out of nothing. It produces it from it's input. By definition, a translation AI is not generative. – Chthonic One Jan 25 '24 at 23:28
  • I did not say the definition in that paper is wrong, quite the opposite. I only said it is most probably not telling us what the common usage in the field is, because it is just one paper with a specific context. – Doc Brown Jan 25 '24 at 23:29
  • Ok, think that new answer is a lot better that your first since it offers a much deeper insight in what counts as generative AI in the field, thanks for that. Still it leaves me puzzled what this precisely means for my title question. – Doc Brown Jan 26 '24 at 23:34
  • I'm glad I could help. It took a while to figure out where our disconnects were. Sometimes when you learn a field you forget how to talk to people who haven't learned it yet, you know? – Chthonic One Jan 28 '24 at 06:46
-1

No, they are not "generative".

As clearly explained by Wicket on the GenAI Beta site, sourced from SirBenet's answer on the same site, Google Translate isn't "generative":

output could be considered too tightly defined by the input text

These translation tools don't generate new text. They translate.
Sure, they use AI to infer meaning to result in better translations, but you can't tell them to "Translate Macbeth to Swahili"...

You'll never get more out of the tools than you put into them.

Cerbrus
  • 115
  • 4