5

I would like to build a domain specific corpus. By this I mean a corpus from a specific knowledge area, for instance Law, Psychiatry, or Klingon language.

I am currently using a Wikipedia corpus, and therefore, I get results that are not very relevant for my domain.

Thank you very much in advance.

Improve this question

3 Answers 3

Reset to default
1

I doubt that this can be fully automated; you will always have to sift through your texts and decide whether they are meeting your criteria for topicality.

You can try to scrape specialised sources (journals or websites only featuring the topic of your interest), but than you run into problems with intellectual property rights: You cannot share your corpus for reproducible research or some kind of contest without the consent of the rights' holders.

Improve this answer
2
  • Thanks for your reply. I have already considered what you say. However, most of websites have a system of "captchas" to prevent massive mining from users
    – Jorgemar
    Dec 19, 2016 at 11:37
  • 2
    There are specialised wikis out there, where the texts are under free licences and downloadable without captachs. Just be sure to follow good practice for crawlers (i.e., crawl "slowly")
    – jk - Reinstate Monica
    Dec 21, 2016 at 17:35
1

You are looking into implementing probably three strategic pillars for your corpus. Machine learning has given computational linguists great tools, and one that you are probably needing is a plain-vanilla bag-of-word algorithm (https://en.wikipedia.org/wiki/Bag-of-words_model).

Python is usually a popular choice. Take a look at:

Then you have semantic trees / databases - such as WordNet (https://wordnet.princeton.edu/wordnet/) that help you extend your corpus based on data already mined for you.

Third, manual review. Lots of it. Your Bag of words will need to be trimmed seriously if it is to be truly useful. The right and wrong branches of semantic trees will also need to be pruned. In any case, this is labour intensive and may require serious expertise in certain fields.

Having said that - you may be better of purchasing some of these corpora.

Improve this answer
0

I'd stick to Wikipedia, and, for example, for the European Union, I'd get all articles in https://en.wikipedia.org/wiki/Category:European_Union (recursing in sub-categories), though that may be too small a corpus for your needs.

(I have been considering doing exactly that, though haven't implemented it yet)

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.