I have a
pandas DataFrame that contains a column with sentences from different languages (6 languages). The DataFrame also contains a column which states which language the corresponding sentence belongs to. However, a sentence may contain non letter ASCII characters such as
=@# etc.. and words that may not belong to the same language. Even though, it may be written in the same script. All the sentences have been correctly tagged to their respective languages. For an example please refer to the below sentence which, has been marked (tagged) as Spanish;
'¿Vas a venir a la tienda conmigo?+== @loja' #Note that 'loja' is a Portuguese word.
Since the sentence is marked as Spanish I would like to remove all non Spanish words and non punctuation characters (+, =, =, @).
I have an idea to remove the non punctuation words by getting the set values and removing the ones that are not letters (there are only few punctuation characters. so no need to search). However, would someone be able to help remove the words that do not belong to the tagged language such as the Portuguese word in the above example using python.?
Thanks & Best Regards