Hot answers tagged

11 votes
Accepted

How can I find the frequency for the Russian word "чем-нибудь"?

Asking the Leipzig Wortschatz Projekt gives the answer: Word: чем-нибудь Number of occurrences: 5,018 Rank: 33,962 Frequency class: 14 Information on the corpus: Information on corpus: Russian Mixed ...
user avatar
9 votes

How can I find the frequency for the Russian word "чем-нибудь"?

Using the Araneum Russicum III corpus (I am linking to the readily available 125Mtokens subcorpus; after a (free) registration you get access to the whole huge 19Gtokens one; the results are from the ...
user avatar
8 votes
Accepted

Conversational English corpus for download

There are many spoken English corpora available. But generally, you need to ask more questions than 'plain text' before you find the right one. Length, level of annotation, format of annotation, type ...
user avatar
8 votes

How to best clean a large historical corpus ridden with OCR errors

For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical ...
user avatar
7 votes
Accepted

What are the conventions for corpus usage?

I can only speak for Germany, and IANAL (I am not a lawyer). The situation is basically as follows: You can collect material from accessible sources (from the web, from radio broadcasts, from TV) and ...
user avatar
6 votes

Looking for Spanish email or chat corpora

The Grupo de Ingeniería Lingüística, part of the Universidad Nacional Autónoma de México has a WhatsApp corpus of undergraduate students. They have a paper that introduces it: http://www.aclweb.org/...
user avatar
  • 450
5 votes

Is there a computational method to syllabify English words?

I looked at the SO discussion (and admit that I can't compute the consequences of all of the code). Those guys even admit that hyphenation solutions can't handle quasi-novel data (such as names of ...
user avatar
  • 69k
5 votes
Accepted

Parts-of-speech tagging and finding relevant phrases in documents

[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ] You ...
user avatar
5 votes
Accepted

glossary/dictionary corpus for NLP task

Two resources are particularly popular. WordNet Wiktionary
user avatar
  • 3,537
5 votes

Word length as a function of word frequency?

Whoops, didn't know I was allowed to answer my own question. Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely ...
user avatar
  • 724
5 votes
Accepted

How to interpret this form of Heaps' Law?

A straightforward rewriting of the Wikipedia formula gives log V_R(n) = log K*n^beta = log K + log n^beta = log K + beta*log n This allows us to identify K=C and beta=-alpha (...
user avatar
5 votes
Accepted

Corpus of English sentences written by non-native speakers?

Learner corpus It seems that you're looking for what's commonly called "learner corpus", i.e. data that's written by people of various skill who don't speak the target language natively but are at ...
user avatar
  • 424
5 votes

How can I find the frequency for the Russian word "чем-нибудь"?

The online version of Russian Nation Corpus (in Russian) offers frequency statistics of word forms, broken down by year: https://processing.ruscorpora.ru/graphic.xml?env=alpha&mode=graphic_main&...
user avatar
  • 689
4 votes

Tool for building a corpus by crawling the web?

There are 3 steps in doing this task: Identifying the languages in webpages Building a crawler that downloads the web page Doing linguistics analysis since my primary NLP language is python and ...
user avatar
  • 1,229
4 votes

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Ironically, your best bet is: https://archive.org/download/stackexchange
user avatar
4 votes

Biggest freely available English corpus?

Sketch Engine, a corpus manager and text analysis software, provides a few corpora with open access for research on https://app.sketchengine.eu/#open The largest English corpus (freely available) is ...
user avatar
  • 91
4 votes
Accepted

Sounds that humans make with their mouths

You may find some useful ideas in the following list: burp Also belch. To expel gas noisily from the stomach through the mouth The baby burped after being fed. cough To push ...
user avatar
  • 309
4 votes

Sounds that humans make with their mouths

First, there are few sounds that people can make with only their mouths (to be more precise, the oral cavity). Clicks, yes, but not breathing, because breathing also involves the lungs. Let's suppose ...
user avatar
  • 69k
4 votes
Accepted

Which free resource to find word frequency?

Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word ...
user avatar
4 votes

JSON for corpora

I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours vertical format: One token per line with ...
user avatar
4 votes
Accepted

How can I know if a log-likelihood score is high enough?

For the purpose of normalisation, you should use a formula such as this: normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000 This formula can be adjusted to normalise, ...
user avatar
  • 4,249
4 votes

How order of the syntax tree is formally/strictly proven?

Good question. Constituency is the theory behind such a tree diagram. There are a bunch of different Constituency tests which you can do on paper: [ [John] [ [hit] [ [ [the] ] [ball] ] ] ] Wh-...
user avatar
  • 638
4 votes

How to best clean a large historical corpus ridden with OCR errors

ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this. (The main reason people do that is because they're about to translate it into 100 ...
user avatar
4 votes

Can someone explain to me the Zipf–Mandelbrot law?

As this is the Linguistics site, I'm guessing it is Zipf's Law you are interested in? If you analyze a corpus of text, and count the number of times each word occurs, you get their frequency. You then ...
user avatar
4 votes

Can someone share with me an article that studies the decline of by-phrases in the passive?

This thesis studies the history of passives and as part of the study looks at the instance of by-phrases, in the journal American journal of botany, grouping articles into three historical clusters – ...
user avatar
  • 69k
3 votes

How should grammatical errors be treated in parsing and tokenisation?

Science must be reproducible, and closely connected to experimental data. This has consequences on scientific practice, and in particular on the how experimental resources and copora should be dealt ...
user avatar
  • 1,497
3 votes

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Yes, there are such corpora. One freely availble corpus is IULA Spanish-English Technical Corpus from Universitat Pompeu Fabra. Another one, concentrating on Computational Linguistcs, is the ACL ...
user avatar
3 votes

Time annotated corpus: plain text english corpus

There is a corpus of internet discussion boards that is split up into weekly files that spans 4 years. It is freely downloadable here: http://www.psych.ualberta.ca/~westburylab/downloads/usenetcorpus....
user avatar
  • 31
3 votes

Lists of linguistic resources

List of resources PHOIBLE phonological database Surrey Morphology Group databases Konstanz Universals Archive Encyclopedia of Linguistic Laws and the Laws in Quantitative Linguistics Glottolog
3 votes

Evaluate idea to autobuild russian-english parallel corpus

There is Russian Learner Translator Corpus which is constructed from students' translations. Maybe it can be useful for you. It has new interface which is in beta now, so there isn't a link to "About" ...
user avatar
  • 581

Only top scored, non community-wiki answers of a minimum length are eligible