How can I do data augmentation for text classification?

Question

I am working on a text classification project. Some properties of my data set:

• Total number of samples: 33200
• Total number of classes: 1131
• Class distribution is highly skewed (about 35% of the classes have only one example)

First of all, it's a highly multi-class classification problem as you can see. Secondly, the data set is not very big (so I don't want to throw away any data). Last but not least, significant portion of the distinct classes have one example. I want to synthesize some more sample data for these classes. So I was searching for data augmentation techniques. However, unlike computer vision, NLP does not seem to have many popular techniques in this regard. Synonym insertion can be one way to do it. Can anybody suggest some other techniques? Thank you!

Answer 1

NoiseMix is made for exactly this. (Full disclosure: I am advisor to the project.)

It applies simple transformations to rows of text data while preserving labels.

The word-level perturbations are: add_letter, repeat_letter, remove_letter, lowercase, remove_punct, word_swap, char_swap, flip_letters and typo_qwerty. The sentence-level perturbations are: remove_space and flip_words

It is still under construction, but the initial benchmarks included the StackExchange question tagging task from the fastText supervised tutorial with about 10000 training rows with 734 labels - similar to your task.

To avoid false precision and a massive table and the nuances of precision and recall for multi-label, I will not quote exact results, but the improvements across tasks are significant, between 3% and 10%, reducing errors by one tenth to almost one half. The exact results will depend on the parameter values you set in config.json.

Noisification is not as effective when the dataset is already large and noisy, and of course training time increases are not a joke at scale. The optimist way to spin diminishing returns is that noisification tends to be more effective on tasks where the dataset size is very small and the initial baseline results very bad.

To get the NoiseMix code and required libs:

git clone https://github.com/noisemix/noisemix.git
cd noisemix
pip install -r requirements.txt

If you have data in fastText format , then you can simply run:

python noisemix.py train.ft.txt -format fastText

This will output a file train.ft.txt.nmx in the same format but with one generated line from every original line, doubling the dataset size.

To increase that, add -versions 2 to get two new lines from every original. But the intuition from the initial results is that it is just as effective to increase the perturbations per line (MAX_PERTURBATION in the config) as to increase the number of lines. The lines look totally butchered, but it works, and trains faster.

The only supported format is fastText (labels prefixed with __label__), you can add other formats in formats.py. You can add data for locales besides English / QWERTY in data.py. Python 3.

You can also try using pre-trained vectors. Conceptually realistic data augmentation is not too different, NoiseMix is just a bit more tuned for user-generated data, whereas fastText Wikipedia pre-trained vectors are a model of the standard formal language.

Answer 2

A method for introducing variation in language data is round-trip translation to a different language and back. Shalom Lappin used machine translation for this purpose (and noted that Google translate is already too good for this purpose and he had to resort to a 2007 version of MOSES).

Answer 3

Compared to images, data augmentation in NLP is tricky as even a slight change in the sentence can entirely change its meaning.

However, there are some augmentation techniques I have seen in the existing literature. These try their best to not change the meaning of the sentence after augmentation.

• Synonym Replacement:
  Replace words in sentences with their synonyms via word embeddings and WordNet database.

• Masked Language Modeling:
  Use BERT, ALBERT, ROBERTA language models to predict some randomly masked part of your original text. This will generate a new sentence.

• Back-translation:
  Translate sentence to another language(e.g. French) and translate it back. You will get a new sentence that retains the original meaning.

• Text Surface Transformation:
  Expand or contract verbal forms. For example, “I am here” -> “I’m here”

• Noise Injection:
  Inject noise into the text such as spelling errors, keyword error simulation, unigram noising, blank noising, sentence shuffling, random word insertion/swapping/deletion.

For an in-depth explanation of each technique, you can refer to this survey on Text Augmentation.