1

I am writing a tokenizer for a small NLP project. After the first attempt (basically splitting at white spaces) didn't lead to good results at all, I am now starting from scratch.

I would like to be able to detect names like "San Francisco" or "New York" as one token. My best guess is that I would need a list to check against for that? Does such a list exist somewhere on the internet?

Improve this question
1
  • Such lists are called gazetteers, and they are sometimes used for NER. However, note that NER is a well-researched field, and there is plenty of literature on the pitfalls of using gazetteers, work arounds, techniques that use them for bootstrapping, and so on. If you are not doing research in the field of NER, why not use some freely available NER?
    – prash ♦
    Commented Jun 11, 2016 at 19:59

1 Answer 1

Reset to default
2

If you're looking for a list of places that you can use for NER, Geonames is a good resource. The GeoNames geographical database covers all countries and contains over eight million placenames that are available for download free of charge.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.