1

Is is possible to tokenize a text in tokens such that first and last name are combined in one token? For example if my text is:

 text = "Barack Obama is the President"

Then:

text.split()

results:

['Barack', 'Obama', 'is', 'the, 'President']

how can I recognize the first and last name? So I get only

['Barack Obama', 'is', 'the', 'President'] 

as tokens.

Is there a way to achieve it in Python?

4

No, going by definition of the word "token". What you're looking for is either a "chunker" or a "Named Entity Recognizer". Some of these are written in Python, and some are not. But that should not matter, as you will be using the output of one of these in your (Python) pipeline.

2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.