11
votes
Accepted
How can I find the frequency for the Russian word "чем-нибудь"?
Asking the Leipzig Wortschatz Projekt gives the answer:
Word: чем-нибудь Number of occurrences: 5,018 Rank: 33,962
Frequency class: 14
Information on the corpus:
Information on corpus:
Russian Mixed ...
- 26.4k
9
votes
How can I find the frequency for the Russian word "чем-нибудь"?
Using the Araneum Russicum III corpus (I am linking to the readily available 125Mtokens subcorpus; after a (free) registration you get access to the whole huge 19Gtokens one; the results are from the ...
- 279
8
votes
Accepted
Conversational English corpus for download
There are many spoken English corpora available. But generally, you need to ask more questions than 'plain text' before you find the right one. Length, level of annotation, format of annotation, type ...
- 10.6k
8
votes
How to best clean a large historical corpus ridden with OCR errors
For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical ...
- 26.4k
7
votes
Accepted
What are the conventions for corpus usage?
I can only speak for Germany, and IANAL (I am not a lawyer). The situation is basically as follows:
You can collect material from accessible sources (from the web, from radio broadcasts, from TV) and ...
- 26.4k
6
votes
Looking for Spanish email or chat corpora
The Grupo de Ingeniería Lingüística, part of the Universidad Nacional Autónoma de México has a WhatsApp corpus of undergraduate students. They have a paper that introduces it: http://www.aclweb.org/...
- 450
5
votes
Is there a computational method to syllabify English words?
I looked at the SO discussion (and admit that I can't compute the consequences of all of the code). Those guys even admit that hyphenation solutions can't handle quasi-novel data (such as names of ...
- 69k
5
votes
Accepted
Parts-of-speech tagging and finding relevant phrases in documents
[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ]
You ...
- 6,972
5
votes
Accepted
glossary/dictionary corpus for NLP task
Two resources are particularly popular.
WordNet
Wiktionary
- 3,537
5
votes
Word length as a function of word frequency?
Whoops, didn't know I was allowed to answer my own question.
Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely ...
- 724
5
votes
Accepted
How to interpret this form of Heaps' Law?
A straightforward rewriting of the Wikipedia formula gives
log V_R(n) = log K*n^beta
= log K + log n^beta
= log K + beta*log n
This allows us to identify K=C and beta=-alpha (...
- 26.4k
5
votes
Accepted
Corpus of English sentences written by non-native speakers?
Learner corpus
It seems that you're looking for what's commonly called "learner corpus", i.e. data that's written by people of various skill who don't speak the target language natively but are at ...
- 424
5
votes
How can I find the frequency for the Russian word "чем-нибудь"?
The online version of Russian Nation Corpus (in Russian) offers frequency statistics of word forms, broken down by year:
https://processing.ruscorpora.ru/graphic.xml?env=alpha&mode=graphic_main&...
- 689
4
votes
Tool for building a corpus by crawling the web?
There are 3 steps in doing this task:
Identifying the languages in webpages
Building a crawler that downloads the web page
Doing linguistics analysis
since my primary NLP language is python and ...
- 1,229
4
votes
Is there any corpus for technical English? (E.g., computers, IT, modern technology)
Ironically, your best bet is: https://archive.org/download/stackexchange
- 654
4
votes
Biggest freely available English corpus?
Sketch Engine, a corpus manager and text analysis software, provides a few corpora with open access for research on https://app.sketchengine.eu/#open The largest English corpus (freely available) is ...
- 91
4
votes
Accepted
Sounds that humans make with their mouths
You may find some useful ideas in the following list:
burp
Also belch. To expel gas noisily from the stomach through the mouth
The baby burped after being fed.
cough
To push ...
- 309
4
votes
Sounds that humans make with their mouths
First, there are few sounds that people can make with only their mouths (to be more precise, the oral cavity). Clicks, yes, but not breathing, because breathing also involves the lungs. Let's suppose ...
- 69k
4
votes
Accepted
Which free resource to find word frequency?
Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word ...
- 26.4k
4
votes
JSON for corpora
I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours
vertical format: One token per line with ...
- 26.4k
4
votes
Accepted
How can I know if a log-likelihood score is high enough?
For the purpose of normalisation, you should use a formula such as this:
normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000
This formula can be adjusted to normalise, ...
- 4,249
4
votes
How order of the syntax tree is formally/strictly proven?
Good question. Constituency is the theory behind such a tree diagram. There are a bunch of different Constituency tests which you can do on paper:
[ [John] [ [hit] [ [ [the] ] [ball] ] ] ]
Wh-...
- 638
4
votes
How to best clean a large historical corpus ridden with OCR errors
ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this.
(The main reason people do that is because they're about to translate it into 100 ...
- 6,972
4
votes
Can someone explain to me the Zipf–Mandelbrot law?
As this is the Linguistics site, I'm guessing it is Zipf's Law you are interested in? If you analyze a corpus of text, and count the number of times each word occurs, you get their frequency. You then ...
- 159
4
votes
Can someone share with me an article that studies the decline of by-phrases in the passive?
This thesis studies the history of passives and as part of the study looks at the instance of by-phrases, in the journal American journal of botany, grouping articles into three historical clusters – ...
- 69k
3
votes
How should grammatical errors be treated in parsing and tokenisation?
Science must be reproducible, and closely connected to experimental data. This has consequences on scientific
practice, and in particular on the how experimental resources and
copora should be dealt ...
- 1,497
3
votes
Is there any corpus for technical English? (E.g., computers, IT, modern technology)
Yes, there are such corpora. One freely availble corpus is IULA Spanish-English Technical Corpus from Universitat Pompeu Fabra.
Another one, concentrating on Computational Linguistcs, is the ACL ...
- 26.4k
3
votes
Time annotated corpus: plain text english corpus
There is a corpus of internet discussion boards that is split up into weekly files that spans 4 years. It is freely downloadable here: http://www.psych.ualberta.ca/~westburylab/downloads/usenetcorpus....
- 31
3
votes
Lists of linguistic resources
List of resources
PHOIBLE phonological database
Surrey Morphology Group databases
Konstanz Universals Archive
Encyclopedia of Linguistic Laws and the Laws in Quantitative Linguistics
Glottolog
3
votes
Evaluate idea to autobuild russian-english parallel corpus
There is Russian Learner Translator Corpus which is constructed from students' translations. Maybe it can be useful for you. It has new interface which is in beta now, so there isn't a link to "About" ...
- 581
Only top scored, non community-wiki answers of a minimum length are eligible