Hot answers tagged

11 votes
Accepted

How can I find the frequency for the Russian word "чем-нибудь"?

Asking the Leipzig Wortschatz Projekt gives the answer: Word: чем-нибудь Number of occurrences: 5,018 Rank: 33,962 Frequency class: 14 Information on the corpus: Information on corpus: Russian Mixed ...
Sir Cornflakes's user avatar
9 votes

How to best clean a large historical corpus ridden with OCR errors

For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical ...
Sir Cornflakes's user avatar
9 votes

How can I find the frequency for the Russian word "чем-нибудь"?

Using the Araneum Russicum III corpus (I am linking to the readily available 125Mtokens subcorpus; after a (free) registration you get access to the whole huge 19Gtokens one; the results are from the ...
Radovan Garabík's user avatar
7 votes
Accepted

What are the conventions for corpus usage?

I can only speak for Germany, and IANAL (I am not a lawyer). The situation is basically as follows: You can collect material from accessible sources (from the web, from radio broadcasts, from TV) and ...
Sir Cornflakes's user avatar
6 votes

Looking for Spanish email or chat corpora

The Grupo de Ingeniería Lingüística, part of the Universidad Nacional Autónoma de México has a WhatsApp corpus of undergraduate students. They have a paper that introduces it: http://www.aclweb.org/...
axme100's user avatar
  • 460
5 votes

Lists of linguistic resources

List of resources PHOIBLE phonological database Surrey Morphology Group databases Konstanz Universals Archive Encyclopedia of Linguistic Laws and the Laws in Quantitative Linguistics Glottolog
5 votes
Accepted

Parts-of-speech tagging and finding relevant phrases in documents

[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ] You ...
Adam Bittlingmayer's user avatar
5 votes
Accepted

glossary/dictionary corpus for NLP task

Two resources are particularly popular. WordNet Wiktionary
prash's user avatar
  • 3,649
5 votes

Word length as a function of word frequency?

Whoops, didn't know I was allowed to answer my own question. Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely ...
Khove's user avatar
  • 764
5 votes
Accepted

How to interpret this form of Heaps' Law?

A straightforward rewriting of the Wikipedia formula gives log V_R(n) = log K*n^beta = log K + log n^beta = log K + beta*log n This allows us to identify K=C and beta=-alpha (...
Sir Cornflakes's user avatar
5 votes
Accepted

Corpus of English sentences written by non-native speakers?

Learner corpus It seems that you're looking for what's commonly called "learner corpus", i.e. data that's written by people of various skill who don't speak the target language natively but are at ...
Peteris's user avatar
  • 414
5 votes

How to best clean a large historical corpus ridden with OCR errors

ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this. (The main reason people do that is because they're about to translate it into 100 ...
Adam Bittlingmayer's user avatar
5 votes

Can someone explain to me the Zipf–Mandelbrot law?

As this is the Linguistics site, I'm guessing it is Zipf's Law you are interested in? If you analyze a corpus of text, and count the number of times each word occurs, you get their frequency. You then ...
Darren Cook's user avatar
5 votes

How can I find the frequency for the Russian word "чем-нибудь"?

The online version of Russian Nation Corpus (in Russian) offers frequency statistics of word forms, broken down by year: https://processing.ruscorpora.ru/graphic.xml?env=alpha&mode=graphic_main&...
Quassnoi's user avatar
  • 699
4 votes

Biggest freely available English corpus?

Sketch Engine, a corpus manager and text analysis software, provides a few corpora with open access for research on https://app.sketchengine.eu/#open The largest English corpus (freely available) is ...
Rodrigo's user avatar
  • 91
4 votes

Tool for building a corpus by crawling the web?

There are 3 steps in doing this task: Identifying the languages in webpages Building a crawler that downloads the web page Doing linguistics analysis since my primary NLP language is python and ...
Andrew Ravus's user avatar
  • 1,265
4 votes
Accepted

Sounds that humans make with their mouths

You may find some useful ideas in the following list: burp Also belch. To expel gas noisily from the stomach through the mouth The baby burped after being fed. cough To push ...
Mr. Black's user avatar
  • 319
4 votes

Sounds that humans make with their mouths

First, there are few sounds that people can make with only their mouths (to be more precise, the oral cavity). Clicks, yes, but not breathing, because breathing also involves the lungs. Let's suppose ...
user6726's user avatar
  • 83k
4 votes
Accepted

Corpus Linguistics: How do I compare date from two corpora correctly?

You could simply divide by the size, so you’re comparing relative frequency. You could also express the log relative frequency, given how small the resulting decimal number is. It’s common to ...
Jeremy Needle's user avatar
4 votes
Accepted

Which free resource to find word frequency?

Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word ...
Sir Cornflakes's user avatar
4 votes

JSON for corpora

I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours vertical format: One token per line with ...
Sir Cornflakes's user avatar
4 votes
Accepted

How can I know if a log-likelihood score is high enough?

For the purpose of normalisation, you should use a formula such as this: normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000 This formula can be adjusted to normalise, ...
robert's user avatar
  • 4,279
4 votes

How order of the syntax tree is formally/strictly proven?

Good question. Constituency is the theory behind such a tree diagram. There are a bunch of different Constituency tests which you can do on paper: [ [John] [ [hit] [ [ [the] ] [ball] ] ] ] Wh-...
purlupar's user avatar
  • 638
4 votes

Can someone share with me an article that studies the decline of by-phrases in the passive?

This thesis studies the history of passives and as part of the study looks at the instance of by-phrases, in the journal American journal of botany, grouping articles into three historical clusters – ...
user6726's user avatar
  • 83k
4 votes

Are linguistic corpora under threat from AI-generated text?

There is a high probability that AI-generated text already exists in web-based corpora (and will increase over time), and a very low probability that such text will appear in e.g. The Corpus of ...
user6726's user avatar
  • 83k
3 votes

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Yes, there are such corpora. One freely availble corpus is IULA Spanish-English Technical Corpus from Universitat Pompeu Fabra. Another one, concentrating on Computational Linguistcs, is the ACL ...
Sir Cornflakes's user avatar
3 votes

Lists of linguistic resources

CLARIN (European Research Infrastructure for Language Resources and Technology) entry points: Virtual Language Observatory (VLO): A search engine for language resources TeLeMaCo: Teaching and ...
3 votes

Where to find frequency list of English words from newspapers, books and magazines?

Expanding on Otavio Macedo's answer, if you just want plain txt files containing English words sorted by frequency without having to analyze Google Ngram Viewer's dataset yourself, Natural Language ...
Flux's user avatar
  • 131
3 votes

Where to find frequency list of English words from newspapers, books and magazines?

This question hasn't been updated in a while. Google also released a 1 Billion Word Benchmark Corpus: https://github.com/tensorflow/models/tree/master/research/lm_1b. It has a vocabulary size of about ...
Adam_G's user avatar
  • 576

Only top scored, non community-wiki answers of a minimum length are eligible