Often I want to make my own corpus of a new language I've become interested in.

I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain text random samples from a specified language edition of Wikipedia and it has worked pretty well.

But sometimes a language doesn't have its own Wikipedia, or its Wikipedia is too small or shows too many artefacts being heavy on articles on certain topics.

What I would like is a tool that crawl the web and gather pages using only a certain language.

It doesn't have to do anything linguistic, raw HTML is usable, plain Unicode text is better, but if it can also do things like word frequency, normalizing, lemmatizing, etc that would be a great bonus.

My current language of interest is Mongolian written in the traditional script, which does indeed seem to have a large enough web presence.

Is there a tool that could help me with this?

  • There is no a complete tool to recognize the language of a text, but you can use dictionary APIs to achieve that goal. Google has a dictionary API, but it seems it is paid. I did not try, but it can be free to a limit (for instance, 300 queries/month). It will help recognizing the language of a text. On the other hand, you can use regular expression and collections library of any programming language to get frequency, to normalize etc.
    – Eray Erdin
    Aug 3, 2014 at 9:49
  • The language recognition wasn't the main challenge, it was the crawling. Aug 3, 2014 at 12:53
  • I think you will get more, and possibly better, answers on stackoverflow or maybe on programmers.stackexchange.com. I would vote to close this question as off-topic as it's about programming, not linguistics.
    – tsuma534
    Mar 3, 2016 at 18:49
  • 1
    I would say datascience.stackexchange.com or opendata.stackexchange.com. Mar 3, 2016 at 19:14

3 Answers 3


There are 3 steps in doing this task:

  1. Identifying the languages in webpages
  2. Building a crawler that downloads the web page
  3. Doing linguistics analysis

since my primary NLP language is python and there are a lot of NLP libraries written for Python we use Python here.

For identifying the language you can use some great language identifiers like this (based on Google's language-detection) and this (Based on guesslanguage.cpp by Jacob R Rideout).

The second step is rather tricky. It's actually called Web Scraping, you can read some great tutorials on web scraping here and here (Scrapy).

For the last step you use different snippets for concordances based on NLTK at here. Other things like word frequency etc. can be used easily via NLTK library.


Although you are asking for a tool to do it yourself, you should not neglect the existence of corpora in many languages on the web, e.g., the Leipzig Wortschatz project at http://corpora.informatik.uni-leipzig.de/ that features 219 languages.

  • I'm talking about languages like Tai Lue from Southern Yunnan, China. I'll be pleasantly surprised if that's among the ones at Leipzig Wortschatz but I'm going to take a look. Thanks. Mar 6, 2016 at 14:18

You can use : https://the.sketchengine.co.uk/auth/corpora/ It's practical and at first, it makes you understand what's what over there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.