I am writing a tokenizer for a small NLP project. After the first attempt (basically splitting at white spaces) didn't lead to good results at all, I am now starting from scratch.

I would like to be able to detect names like "San Francisco" or "New York" as one token. My best guess is that I would need a list to check against for that? Does such a list exist somewhere on the internet?

  • Such lists are called gazetteers, and they are sometimes used for NER. However, note that NER is a well-researched field, and there is plenty of literature on the pitfalls of using gazetteers, work arounds, techniques that use them for bootstrapping, and so on. If you are not doing research in the field of NER, why not use some freely available NER? – prash Jun 11 '16 at 19:59

If you're looking for a list of places that you can use for NER, Geonames is a good resource. The GeoNames geographical database covers all countries and contains over eight million placenames that are available for download free of charge.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.