I have a large excel file like the following:
Timestamp Text Work Id 5/4/16 17:52 rain a lot the packs maybe damage. Delivery XYZ 5/4/16 18:29 wh. screen Other ABC 5/4/16 14:54 15107 Lane Pflugerville, TX customer called me and his phone number and my phone numbers were not masked. thank you customer has had a stroke and items were missing from his delivery the cleaning supplies for his wet vacuum steam cleaner. he needs a call back from customer support Delivery YYY 5/6/16 13:05 How will I know if I Signing up ASX 5/4/16 23:07 an quality Delivery DFC
I want to work only on the "Text" column and then eliminate those row that have basically just have gibberish in the "Text" column (rows 2,4,5 from the above example).
I'm reading only the 2nd column as follow:
import xlrd book = xlrd.open_workbook("excel.xlsx") sheet = book.sheet_by_index(0) for row_index in xrange(1, sheet.nrows): # skip heading row timestamp, text = sheet.row_values(row_index, end_colx=2) text) print (text)
How do I remove the gibberish rows? I have an idea that I need to work with
nltk and have a positive corpus (one that does not have any gibberish), one negative corpus (only having gibberish text), and train my model with it. But how do I go about implementing it? Please help!!