I am working on a text classification project. Some properties of my data set:

  • Total number of samples: 33200
  • Total number of classes: 1131
  • Class distribution is highly skewed (about 35% of the classes have only one example)

First of all, it's a highly multi-class classification problem as you can see. Secondly, the data set is not very big (so I don't want to throw away any data). Last but not least, significant portion of the distinct classes have one example. I want to synthesize some more sample data for these classes. So I was searching for data augmentation techniques. However, unlike computer vision, NLP does not seem to have many popular techniques in this regard. Synonym insertion can be one way to do it. Can anybody suggest some other techniques? Thank you!

  • 1
    I just answered with a technique for your specific problem, so you can move forward. But What type of data augmentation techniques are used for NLP? is a great question, let's make a new question for it. – Adam Bittlingmayer May 25 '18 at 6:34
  • 1
    If you don't mind, I changed the title now to match the body. – Adam Bittlingmayer May 25 '18 at 7:58

NoiseMix is made for exactly this. (Full disclosure: I am advisor to the project.)

It applies simple transformations to rows of text data while preserving labels.

The word-level perturbations are: add_letter, repeat_letter, remove_letter, lowercase, remove_punct, word_swap, char_swap, flip_letters and typo_qwerty. The sentence-level perturbations are: remove_space and flip_words

It is still under construction, but the initial benchmarks included the StackExchange question tagging task from the fastText supervised tutorial with about 10000 training rows with 734 labels - similar to your task.

To avoid false precision and a massive table and the nuances of precision and recall for multi-label, I will not quote exact results, but the improvements across tasks are significant, between 3% and 10%, reducing errors by one tenth to almost one half. The exact results will depend on the parameter values you set in config.json.

Noisification is not as effective when the dataset is already large and noisy, and of course training time increases are not a joke at scale. The optimist way to spin diminishing returns is that noisification tends to be more effective on tasks where the dataset size is very small and the initial baseline results very bad.

To get the NoiseMix code and required libs:

git clone https://github.com/noisemix/noisemix.git
cd noisemix
pip install -r requirements.txt

If you have data in fastText format , then you can simply run:

python noisemix.py train.ft.txt -format fastText

This will output a file train.ft.txt.nmx in the same format but with one generated line from every original line, doubling the dataset size.

To increase that, add -versions 2 to get two new lines from every original. But the intuition from the initial results is that it is just as effective to increase the perturbations per line (MAX_PERTURBATION in the config) as to increase the number of lines. The lines look totally butchered, but it works, and trains faster.

The only supported format is fastText (labels prefixed with __label__), you can add other formats in formats.py. You can add data for locales besides English / QWERTY in data.py. Python 3.

You can also try using pre-trained vectors. Conceptually realistic data augmentation is not too different, NoiseMix is just a bit more tuned for user-generated data, whereas fastText Wikipedia pre-trained vectors are a model of the standard formal language.

| improve this answer | |
  • 1
    Are the operations guaranteed to produce existent words, or are they just random? (I know that data augmentation in CV has stuff like skewing images and rotating them, which makes more sense to me as 'valid' images, so I was wondering if something similar's going on here or if it's really just adding random noise.) – WavesWashSands May 25 '18 at 7:16
  • 1
    @WavesWashSands It's really just random. Yes, it's a challenge, there are not really invalid images, but there are invalid sentences, and sometimes they differ very little from the valid ones. The space is sparse. This is probably why this set of techniques is not as developed for text as for images. This high-level discussion deserves its own question though. – Adam Bittlingmayer May 25 '18 at 7:52
  • 1
    @WavesWashSands Original: __label__sauce __label__cheese How much does potato starch affect a cheese sauce recipe? Generated: __label__sauce __label__cheese How much does po4aot starch affect a ceese sauce recipe?, __label__sauce __label__cheese Ho much dooes po4aot starch affecta ceesie zaucDe recipe, __label__sauce __label__cheese Ho much dooes po4aot starch affecta TceesiezaucDe recipe – Adam Bittlingmayer May 25 '18 at 7:53
  • 1
    @WavesWashSands The natural evolution of all of this will be a model learned from data, instead of hand-built rules, to generate noisy data from clean data. – Adam Bittlingmayer May 25 '18 at 8:14
  • @A. M. Bittlingmayer which dependencies and their versions are required for Noisemix. I am facing various invalid syntax errors. Moreover does it run with Python 2. – Uzair A. Nov 6 '18 at 10:56

A method for introducing variation in language data is round-trip translation to a different language and back. Shalom Lappin used machine translation for this purpose (and noted that Google translate is already too good for this purpose and he had to resort to a 2007 version of MOSES).

| improve this answer | |

Compared to images, data augmentation in NLP is tricky as even a slight change in the sentence can entirely change its meaning.

However, there are some augmentation techniques I have seen in the existing literature. These try their best to not change the meaning of the sentence after augmentation.

  • Synonym Replacement:
    Replace words in sentences with their synonyms via word embeddings and WordNet database.

  • Masked Language Modeling:
    Use BERT, ALBERT, ROBERTA language models to predict some randomly masked part of your original text. This will generate a new sentence.

  • Back-translation:
    Translate sentence to another language(e.g. French) and translate it back. You will get a new sentence that retains the original meaning.

  • Text Surface Transformation:
    Expand or contract verbal forms. For example, “I am here” -> “I’m here”

  • Noise Injection:
    Inject noise into the text such as spelling errors, keyword error simulation, unigram noising, blank noising, sentence shuffling, random word insertion/swapping/deletion.

For an in-depth explanation of each technique, you can refer to this survey on Text Augmentation.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.