I have a large excel file like the following:

Timestamp       Text                                Work        Id
5/4/16 17:52    rain a lot the packs maybe damage.  Delivery    XYZ
5/4/16 18:29    wh. screen                          Other       ABC
5/4/16 14:54    15107 Lane Pflugerville, 
                TX customer called me and his phone 
                number and my phone numbers were not 
                masked. thank you customer has had a 
                stroke and items were missing from his 
                delivery the cleaning supplies for his 
                wet vacuum steam cleaner.  he needs a 
                call back from customer support     Delivery    YYY
5/6/16 13:05    How will I know if I                Signing up  ASX
5/4/16 23:07    an quality                          Delivery    DFC

I want to work only on the "Text" column and then eliminate those row that have basically just have gibberish in the "Text" column (rows 2,4,5 from the above example).

I'm reading only the 2nd column as follow:

import xlrd
book = xlrd.open_workbook("excel.xlsx")
sheet = book.sheet_by_index(0)
for row_index in xrange(1, sheet.nrows): # skip heading row
    timestamp, text = sheet.row_values(row_index, end_colx=2)
    print (text)

How do I remove the gibberish rows? I have an idea that I need to work with nltk and have a positive corpus (one that does not have any gibberish), one negative corpus (only having gibberish text), and train my model with it. But how do I go about implementing it? Please help!!

  • 2
    I'm voting to close this question as off-topic because it looks like a pure programming question, more suitable at Stack Overflow – bytebuster May 4 '17 at 8:25
  • 1
    None of those Text examples are gibberish. Some of them are simply longer than others (#1 is, for example, not grammatical). I think you’ll have a hard time classifying them via language. – Jeremy Needle May 4 '17 at 11:37
  • 3
    I am voting to leave this question open. It looks like it asks primarily about the conceptual approach to solving the problem, and not about the technical details of the implementation. – lemontree May 4 '17 at 12:22
  • 1
    I agree with @lemontree. Also, if we're going to have an nlp tag, then I think programming questions are appropriate. – Adam_G May 4 '17 at 22:11
  • 2
    I'm voting to close this question as off-topic because this question is mainly about computing and not about linguistics. – jknappen Jul 3 '17 at 21:30

Define “meaningful”. Implement that definition by means of a filter that will accept meaningful text, i.e., text that can be understood under such rules.

A modern way to solve this problem would be to train a neural network of some sort with a large set of phrases that are “meaningful” (e.g., famous quotes and excerpts from books) and a set of randomly-generated “gibberish” strings.

A cheaper option would be to look for specific words in the English language. Adverbs, conjunctions, prepositions and determiners, as there's a finite list of them and they're most likely to be present in colloquial language. Telling nouns, verbs and adjectives apart is much more difficult, as there are words that can be used either way (“answer”, “giant”, etc.)

Neither option will allow you to accept street addresses, phone numbers and other kind of structured data as “meaningful” text; you'll have to parse them separately. In particular, “fifteen thousand one hundred and seven” is meaningful English text, but “15107” is a number.

  • Isn't there an nltk corpus that will have a corpus of English words? – Arman May 5 '17 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.