5 votes
Accepted

Where can I find the letters of documented writing systems, as text, online?

Most well documented writing systems have been encoded in Unicode, at least to some degree. You can see a list of supported scripts here and code charts for each script here. A few scripts (including ...
Uri Granta's user avatar
  • 1,162
3 votes

Is a text with orthographic or grammatic mistakes in a language X still a text in that language X?

A useful philosophical framework for discussion is the classic dichotomy between competence and performance, promulgated by the theory of generative grammar, and the difference by I-language and E-...
user6726's user avatar
  • 83k
3 votes

How can I do data augmentation for text classification?

A method for introducing variation in language data is round-trip translation to a different language and back. Shalom Lappin used machine translation for this purpose (and noted that Google translate ...
Sir Cornflakes's user avatar
3 votes
Accepted

How can I do data augmentation for text classification?

NoiseMix is made for exactly this. (Full disclosure: I am advisor to the project.) It applies simple transformations to rows of text data while preserving labels. The word-level perturbations are:...
Adam Bittlingmayer's user avatar
2 votes

What are the various approaches to detect whether a sentence is complete or not?

For formal English anyway you can parse with spaCy and then iterate through the tagged tokens looking for a finite verb (VerbForm=Fin, as opposed to Ger or Inf). See https://spacy.io/usage/linguistic-...
Adam Bittlingmayer's user avatar
2 votes
Accepted

What parts of linguistics deal with the differences between text types?

The features mentioned in the last paragraph of the question belong to the domain of corpus linguistics and stylometry. For the categories mentioned in the first part of the question, Systemic ...
Sir Cornflakes's user avatar
2 votes

How can I do data augmentation for text classification?

Compared to images, data augmentation in NLP is tricky as even a slight change in the sentence can entirely change its meaning. However, there are some augmentation techniques I have seen in the ...
Amit Chaudhary's user avatar
2 votes

Is a text with orthographic or grammatic mistakes in a language X still a text in that language X?

Yes If the sentence is intended to be in that language, then I'd say that it is in that language. Intent matters a lot in communication - for example, the utterance "Hallo!" would be a valid sentence ...
Peteris's user avatar
  • 414
1 vote

Is a text with orthographic or grammatic mistakes in a language X still a text in that language X?

Yes, of course. In corpus linguistics we often have to deal with texts digitised using OCR and containing some amount of OCR errors. We are completely aware that we cannot get rid of the OCR errors ...
Sir Cornflakes's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible