11

The corpus in NLTK sinica can be used for research.
There are two shortcomings for sinica to be used in research.

  1. too small
  2. it is a traditional Chinese corpus

Can someone here introduce a big and simplified Chinese corpus that can be freely downloaded for research?

2
  • 1
    This is an interesting problem; as this script is a recent innovation, most texts written in Simplified Chinese are by default protected by copyright, and hence can't be used as part of a text corpus. See Article 5 of PRC Copyright Law for exceptions.
    – Locoluis
    Aug 22, 2017 at 13:52
  • The website blah.me has plaintext books numbered from https://blah.me/download/ebook/1000/txt to https://blah.me/download/ebook/2294/txt. It's possible to wget them all, which equates to just over 9 million sentences. Sep 3, 2020 at 12:57

4 Answers 4

4

The Opus corpus (open subtitles) has a chinese subcorpus. It really comes much in handy when you want to train tools with parallel corpora. It is open source and the lannguage is derived from movies so it shouldn't be too traditional. http://opus.lingfil.uu.se/

3

There are some standard choices:

2

A new plain text simplified Chinese corpus CLUECorpus2020 was released this year; see the paper at the arXiv. It is extracted from Common Crawl between the dates of July to December 2019.

The files are downloadable from links from their Github page (which lead to Google or Baidu cloud services). They include 新闻语料 (news corpus) 8GB, 社区互动-语料 (social interaction corpus) 3GB, 维基百科-语料 (Wikipedia corpus) 1.1GB, 评论数据-语料 (comment data corpus) 2.3GB.


The other large corpus I'm aware of is the Leiden Weibo Corpus (download from here) which "consists of 5,103,566 messages posted on Sina Weibo in January 2012" (Weibo is sometimes considered the Chinese equivalent to Twitter). An example from this corpus is:

"3399658661888659","44","3","m","有建要学会快乐","啊~~~裸考...现在还来得及,全裸还是半裸?快点复习一下电脑才行呐...我不要跟师弟师妹一起上课[泪]不不不,好烦好烦[生病][生病][生病]","30","啊 裸考 现在 还 来 得 及 , 全裸 还是 半裸 ? 快点 复习 一下 电脑 才行呐 我 不 要 跟 师弟 师妹 一起 上课不不不 , 好 烦 好 烦","啊/SP 裸考/NN 现在/NT 还/AD 来/VV 得/DER 及/VV ,/PU 全裸/AD 还是/AD 半裸/VV ?/PU 快点/NN 复习/VV 一下/AD 电脑/NN 才行呐/NN 我/PN 不/AD 要/VV 跟/P 师弟/NN 师妹/NN 一起/AD 上课不不不/VV ,/PU 好/AD 烦/VV 好/AD 烦/VV "

This not only includes the plaintext but also segmentation and POS tags and further information.

0

This one is fairly easy to use, although it cannot be downloaded, but only accessed online: http://corpus.leeds.ac.uk/query-zh.html. It consists of three corporas: Chinese Internet Corpus (280 million tokens), Chinese Business Corpus (30 million tokens) and Lancaster Corpus of Mandarin Chinese (more on this one here: http://www.lancaster.ac.uk/fass/projects/corpus/LCMC/).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.