How to extract rows with only meaningful text in a column

Question

I have a large excel file like the following:

Timestamp       Text                                Work        Id
5/4/16 17:52    rain a lot the packs maybe damage.  Delivery    XYZ
5/4/16 18:29    wh. screen                          Other       ABC
5/4/16 14:54    15107 Lane Pflugerville, 
                TX customer called me and his phone 
                number and my phone numbers were not 
                masked. thank you customer has had a 
                stroke and items were missing from his 
                delivery the cleaning supplies for his 
                wet vacuum steam cleaner.  he needs a 
                call back from customer support     Delivery    YYY
5/6/16 13:05    How will I know if I                Signing up  ASX
5/4/16 23:07    an quality                          Delivery    DFC

I want to work only on the "Text" column and then eliminate those row that have basically just have gibberish in the "Text" column (rows 2,4,5 from the above example).

I'm reading only the 2nd column as follow:

import xlrd
book = xlrd.open_workbook("excel.xlsx")
sheet = book.sheet_by_index(0)
for row_index in xrange(1, sheet.nrows): # skip heading row
    timestamp, text = sheet.row_values(row_index, end_colx=2)
    text)
    print (text)

How do I remove the gibberish rows? I have an idea that I need to work with nltk and have a positive corpus (one that does not have any gibberish), one negative corpus (only having gibberish text), and train my model with it. But how do I go about implementing it? Please help!!

Answer 1

Define “meaningful”. Implement that definition by means of a filter that will accept meaningful text, i.e., text that can be understood under such rules.

A modern way to solve this problem would be to train a neural network of some sort with a large set of phrases that are “meaningful” (e.g., famous quotes and excerpts from books) and a set of randomly-generated “gibberish” strings.

A cheaper option would be to look for specific words in the English language. Adverbs, conjunctions, prepositions and determiners, as there's a finite list of them and they're most likely to be present in colloquial language. Telling nouns, verbs and adjectives apart is much more difficult, as there are words that can be used either way (“answer”, “giant”, etc.)

Neither option will allow you to accept street addresses, phone numbers and other kind of structured data as “meaningful” text; you'll have to parse them separately. In particular, “fifteen thousand one hundred and seven” is meaningful English text, but “15107” is a number.