1

I have a pandas DataFrame that contains a column with sentences from different languages (6 languages). The DataFrame also contains a column which states which language the corresponding sentence belongs to. However, a sentence may contain non letter ASCII characters such as =@# etc.. and words that may not belong to the same language. Even though, it may be written in the same script. All the sentences have been correctly tagged to their respective languages. For an example please refer to the below sentence which, has been marked (tagged) as Spanish;

'¿Vas a venir a la tienda conmigo?+== @loja' #Note that 'loja' is a Portuguese word.

Since the sentence is marked as Spanish I would like to remove all non Spanish words and non punctuation characters (+, =, =, @).

I have an idea to remove the non punctuation words by getting the set values and removing the ones that are not letters (there are only few punctuation characters. so no need to search). However, would someone be able to help remove the words that do not belong to the tagged language such as the Portuguese word in the above example using python.?

This question is related to this question asked earlier. I have also asked the same here in Stack Overflow.

Thanks & Best Regards

Michael

Improve this question
2
  • 1
    You can use stopwords if they are a few non Spanish words. Likewise, you can use Spacy that can handle Spanish to tokenize your sentences (spacy.io/models/es#___gatsby)
    – amegnunsen
    Apr 17, 2020 at 9:18
  • Hi, Thanks for reply.
    – Michael Schroter
    Apr 17, 2020 at 9:46

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.