12
Here is a variety of language corpora with millions of sentences each: http://corpora.informatik.uni-leipzig.de/download.html
11
Asking the Leipzig Wortschatz Projekt gives the answer:
Word: чем-нибудь Number of occurrences: 5,018 Rank: 33,962
Frequency class: 14
Information on the corpus:
Information on corpus:
Russian Mixed 2013
Description
Russian mixed corpus based on material from 2013
Details
Name rus_mixed_2013 Sentences 118,550,890
Language Russian () Types 14,388,...
answered Oct 26 '21 at 9:14
jk - Reinstate Monica
24.3k3 gold badges57 silver badges105 bronze badges
10
As with all natural laws, Zipf's law is an approximation. If you take a large corpus, and compute the Zipf curve, it will more or less follow a Zipf distribution (with coefficients thrown in to account for slack).
This doesn't mean that for every language it follows the exact rule of 'the second most common lexical item is 1/2 as frequent as the most common'...
9
In addition to the NPS chat corpus included with NLTK, also see NUS text message corpus for text conversations. You can also use the Twitter API to build a corpus of tweets as well.
8
Zipf’s law, as I understand it, is not really about languages, but about statistics and probability. It is just one of several formulations of the fact that many non-arbitrary sequences of numbers (frequency of words in a given corpus; population size of cites in relation to their rank; annual turnover of ranked companies; etc., etc.) are not evenly ...
8
There are many spoken English corpora available. But generally, you need to ask more questions than 'plain text' before you find the right one. Length, level of annotation, format of annotation, type of conversation, genre/register, dialect, natural vs. elicited, etc. Those will all depend on the type of research questions you want to answer.
If you just ...
8
For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical errors automatically.
We improved on that for the Royal Society Corpus and our scripts to do that are available for download here. Our approach is tuned for ...
answered Feb 5 '21 at 15:08
jk - Reinstate Monica
24.3k3 gold badges57 silver badges105 bronze badges
8
Using the Araneum Russicum III corpus (I am linking to the readily available 125Mtokens subcorpus; after a (free) registration you get access to the whole huge 19Gtokens one; the results are from the 19Gtoken corpus):
что-нибудь (nominative): 25.20 ipm
чем-нибудь (instrumental): 3.50 ipm
ч.*-нибудь (should take care of all the cases, link to the frequency ...
7
I can only speak for Germany, and IANAL (I am not a lawyer). The situation is basically as follows:
You can collect material from accessible sources (from the web, from radio broadcasts, from TV) and do analyses on that material
You can do so within a closed collaboration with some collaborators including students and guests visiting your institution (they ...
answered Jul 5 '19 at 13:15
jk - Reinstate Monica
24.3k3 gold badges57 silver badges105 bronze badges
6
The Grupo de Ingeniería Lingüística, part of the Universidad Nacional Autónoma de México has a WhatsApp corpus of undergraduate students. They have a paper that introduces it: http://www.aclweb.org/anthology/W18-3501.
They have their own web-based corpus management tool http://www.corpus.unam.mx/geco/ but I don't see the WhatsApp corpus on there, so you ...
5
I looked at the SO discussion (and admit that I can't compute the consequences of all of the code). Those guys even admit that hyphenation solutions can't handle quasi-novel data (such as names of Welsh origin). Linguists can contribute linguistic clarity to the discussion.
I maintain that the SO answers do not yield correct syllabification, since ...
5
[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ]
You must break down the problem.
0. Sentence Segmentation and Tokenisation
I'm assuming this is already done.
1. Find named entities
You need a first pass to find ...
5
Two resources are particularly popular.
WordNet
Wiktionary
5
Whoops, didn't know I was allowed to answer my own question.
Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely proportional to its rank in frequency, so the most common word is used twice as much as the second most common word, three times as much as the third most common, and ...
5
A straightforward rewriting of the Wikipedia formula gives
log V_R(n) = log K*n^beta
= log K + log n^beta
= log K + beta*log n
This allows us to identify K=C and beta=-alpha (probably the WSJ uses a different formulation of Heaps' law
V_R (n) = \frac{K}{n^\alpha}
). The remaining b is a strange additional parameter not present in ...
answered Apr 9 '19 at 15:16
jk - Reinstate Monica
24.3k3 gold badges57 silver badges105 bronze badges
5
Learner corpus
It seems that you're looking for what's commonly called "learner corpus", i.e. data that's written by people of various skill who don't speak the target language natively but are at various proficiency levels of learning it.
There are many such resources for English, the largest of which seems to be the Cambridge Learner Corpus, however, it ...
4
There are 3 steps in doing this task:
Identifying the languages in webpages
Building a crawler that downloads the web page
Doing linguistics analysis
since my primary NLP language is python and there are a lot of NLP libraries written for Python we use Python here.
For identifying the language you can use some great language identifiers like this (based ...
4
The IMS Open Corpus Workbench might work for you.
4
Ironically, your best bet is: https://archive.org/download/stackexchange
4
Sketch Engine, a corpus manager and text analysis software, provides a few corpora with open access for research on https://app.sketchengine.eu/#open The largest English corpus (freely available) is ACL Anthology Reference Corpus with 62 million words.
On the other hand, you can try 30-day free trial of Sketch Engine and search one of the biggest English ...
4
You may find some useful ideas in the following list:
burp
Also belch. To expel gas noisily from the stomach through the mouth
The baby burped after being fed.
cough
To push air from the lungs in a quick, noisy explosion.
He started to cough once he had a cold .
hum
To make a sound from the vocal chords without ...
4
First, there are few sounds that people can make with only their mouths (to be more precise, the oral cavity). Clicks, yes, but not breathing, because breathing also involves the lungs. Let's suppose that you mean "the body parts used for speech", but not necessarily in the way that they are used in speech. (Ingressive lung air uses the anatomy, but in a non-...
4
Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word prevalence for 61,800 English words by Marc Brysbaert, Paweł Mandera, Samantha McCormick, and Emmanuel Keuleers in the link given.
answered Jun 21 '18 at 13:23
jk - Reinstate Monica
24.3k3 gold badges57 silver badges105 bronze badges
4
I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours
vertical format: One token per line with annotations separated by TABs
inline format: plain text with annotations appended to each token like this: plain/ADJ text/NOUN with/PREP ... (the seperating character ...
answered Jul 2 '18 at 10:31
jk - Reinstate Monica
24.3k3 gold badges57 silver badges105 bronze badges
4
For the purpose of normalisation, you should use a formula such as this:
normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000
This formula can be adjusted to normalise, for example, per 100,000 words - just replace the 1,000,000 in the formula by 100,000.
When calculating log likelihood for a comparison of the frequency of a ...
4
Good question. Constituency is the theory behind such a tree diagram. There are a bunch of different Constituency tests which you can do on paper:
[ [John] [ [hit] [ [ [the] ] [ball] ] ] ]
Wh-substitution:
[John]: Who hit the ball?
[the]: John hit which ball?
Wh-substitution and do-support-substitution:
[hit]: John did what to the ball?
Importantly, ...
4
ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this.
(The main reason people do that is because they're about to translate it into 100 languages, and would rather fix the errors just once.)
There are two ways to try to do what you want to do.
1. Just pick a dummy target language, like English
Let's ...
4
As this is the Linguistics site, I'm guessing it is Zipf's Law you are interested in? If you analyze a corpus of text, and count the number of times each word occurs, you get their frequency. You then line then up on the x-axis of a graph, in order of rank (i.e. the most common word comes at x=1, the second most common word at x=2, and the least common word ...
4
The online version of Russian Nation Corpus (in Russian) offers frequency statistics of word forms, broken down by year:
https://processing.ruscorpora.ru/graphic.xml?env=alpha&mode=graphic_main&mycorp=&mysent=&mysize=&mysentsize=&mydocsize=&dpp=100&spp=&spd=1&text=lexform&sort=i_year_created&g=i_year_created&...
3
There is a corpus of internet discussion boards that is split up into weekly files that spans 4 years. It is freely downloadable here: http://www.psych.ualberta.ca/~westburylab/downloads/usenetcorpus.download.html
Only top voted, non community-wiki answers of a minimum length are eligible