Are linguistic corpora under threat from AI-generated text?

Question

As many will be aware, there has been a lot of concern about the spread of AI generated text posted on the Stack Exchange network, leading to moderators on several sites standing down. See here and also here.

My question is not about the spread of misinformation as such, but rather whether various corpora such as the Mark Davies corpora including COCA, BNC, iWeb and NOW may be being contaminated by AI generated texts, and if so, what can be done about this.

It is clear that, at the most basic level, we cannot understand the frequency with which certain terms are used or collocate with others in actual human speech or writing if the data actually show how often ChatGPT, for example, produces these terms.

So:

1. Are corpora in general, and the Mark Davies corpora in particular under threat from, or perhaps already contaminated by, AI-produced texts?

2. What, if anything, can be done about this?

---

^{The purpose of mentioning the Stack Exchange issues concerning AI-generated text is just to demonstrate that this is not a hypothetical question about a future phenomenon that may not occur, but is a question about what demonstrably is already happening: i.e. AI generated language appearing where it may be taken for human speech.}

Answer 1

There is a high probability that AI-generated text already exists in web-based corpora (and will increase over time), and a very low probability that such text will appear in e.g. The Corpus of Victorian Poetry.

Whether or not this is a threat depends on what your standards / assumptions are. To begin with, these corpora almost never have anything to do with human speech (there are LDC corpora of actual speech), instead they have to do with (quasi-)linguistic texts constructed by humans – or, now, the chat-bot. Second, SE uses English as its language for Q&A (apart from some language stacks where one can post in Russian or Spanish), but not every person who posts on SE is a native speaker of English, or a fluent speaker, or a good speaker. On occasion, people apparently use Google Translate to generate an English version of their text. I don't assume that "SE English" is a good source of data for estimating the ordinary conversational behavior of English native speakers. Would you say that productions of non-fluent writers of English "threatens" corpora?

I've been exposed to linguistic studies of other (more-obscure) languages based on web scrapings, and have never seen any satisfactory resolution to the problem that you never know if the anonymous author actually speaks that language. The reason why I think The Corpus of Victorian Poetry or the Perseus corpora are not threatened is that they are more selective in their inclusion criteria. I think that Google N-gram is likewise based on more reliable sources (published books), but that does limit one in terms of style.

Answer 2

The Mark Davies' corpora are free of machine generated text with a very high probability. We cannot completely exclude some machine translations (a very specific type of machine generated text) in the News and Web registers, and even that is probably post-edited by a human.

And those corpora are finished and fixed after the point of their publication, so they will stay in their present status permanently.

P.S. Something that wasn't asked for: The Davies' corpora are known to have contaminations from the "other" English: There is some British English in COCA, and there is some American English in the BNC. It is sneaking in for various reasons and I am not aware of a systematic tagging of the "other English" contamination.