Hot answers tagged

22

Corpora containing more than 15 million words are often not freely available due to copyright issues (such as the British National Corpus and the Corpus of Contemporary American English). The open part of the American National Corpus (OANC) might fulfill your criteria. It contains almost 15 m. words, it's free, and contains conversations and other genres. ...


12

Here is a variety of language corpora with millions of sentences each: http://corpora.informatik.uni-leipzig.de/download.html


11

This is a complex topic but here's an attempt at an answer. My background is in describing and documenting relatively small, endangered languages, so I'll describe how it's done in that situation. I'll assume you have computers, software, internet, native speaking informants, recording equipment, and many years to spend on the work. Firstly, recorded texts ...


9

As with all natural laws, Zipf's law is an approximation. If you take a large corpus, and compute the Zipf curve, it will more or less follow a Zipf distribution (with coefficients thrown in to account for slack). This doesn't mean that for every language it follows the exact rule of 'the second most common lexical item is 1/2 as frequent as the most common'...


9

In addition to the NPS chat corpus included with NLTK, also see NUS text message corpus for text conversations. You can also use the Twitter API to build a corpus of tweets as well.


8

Zipf’s law, as I understand it, is not really about languages, but about statistics and probability. It is just one of several formulations of the fact that many non-arbitrary sequences of numbers (frequency of words in a given corpus; population size of cites in relation to their rank; annual turnover of ranked companies; etc., etc.) are not evenly ...


8

There are many spoken English corpora available. But generally, you need to ask more questions than 'plain text' before you find the right one. Length, level of annotation, format of annotation, type of conversation, genre/register, dialect, natural vs. elicited, etc. Those will all depend on the type of research questions you want to answer. If you just ...


8

For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical errors automatically. We improved on that for the Royal Society Corpus and our scripts to do that are available for download here. Our approach is tuned for ...


7

I can only speak for Germany, and IANAL (I am not a lawyer). The situation is basically as follows: You can collect material from accessible sources (from the web, from radio broadcasts, from TV) and do analyses on that material You can do so within a closed collaboration with some collaborators including students and guests visiting your institution (they ...


6

The Grupo de Ingeniería Lingüística, part of the Universidad Nacional Autónoma de México has a WhatsApp corpus of undergraduate students. They have a paper that introduces it: http://www.aclweb.org/anthology/W18-3501. They have their own web-based corpus management tool http://www.corpus.unam.mx/geco/ but I don't see the WhatsApp corpus on there, so you ...


5

I looked at the SO discussion (and admit that I can't compute the consequences of all of the code). Those guys even admit that hyphenation solutions can't handle quasi-novel data (such as names of Welsh origin). Linguists can contribute linguistic clarity to the discussion. I maintain that the SO answers do not yield correct syllabification, since ...


5

[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ] You must break down the problem. 0. Sentence Segmentation and Tokenisation I'm assuming this is already done. 1. Find named entities You need a first pass to find ...


5

Two resources are particularly popular. WordNet Wiktionary


5

A straightforward rewriting of the Wikipedia formula gives log V_R(n) = log K*n^beta = log K + log n^beta = log K + beta*log n This allows us to identify K=C and beta=-alpha (probably the WSJ uses a different formulation of Heaps' law V_R (n) = \frac{K}{n^\alpha} ). The remaining b is a strange additional parameter not present in ...


5

Learner corpus It seems that you're looking for what's commonly called "learner corpus", i.e. data that's written by people of various skill who don't speak the target language natively but are at various proficiency levels of learning it. There are many such resources for English, the largest of which seems to be the Cambridge Learner Corpus, however, it ...


4

There are 3 steps in doing this task: Identifying the languages in webpages Building a crawler that downloads the web page Doing linguistics analysis since my primary NLP language is python and there are a lot of NLP libraries written for Python we use Python here. For identifying the language you can use some great language identifiers like this (based ...


4

In English, you're fine with apostrophes. Compounds with hyphens, such as ice-cream, are mostly treated as separated tokens in corpora. As for other languages in Europe, the "long" L in Catalan is denoted by a "bullet", e.g. pel·lícula.


4

The IMS Open Corpus Workbench might work for you.


4

Ironically, your best bet is: https://archive.org/download/stackexchange


4

Sketch Engine, a corpus manager and text analysis software, provides a few corpora with open access for research on https://app.sketchengine.eu/#open The largest English corpus (freely available) is ACL Anthology Reference Corpus with 62 million words. On the other hand, you can try 30-day free trial of Sketch Engine and search one of the biggest English ...


4

You may find some useful ideas in the following list: burp Also belch. To expel gas noisily from the stomach through the mouth The baby burped after being fed. cough To push air from the lungs in a quick, noisy explosion. He started to cough once he had a cold . hum To make a sound from the vocal chords without ...


4

First, there are few sounds that people can make with only their mouths (to be more precise, the oral cavity). Clicks, yes, but not breathing, because breathing also involves the lungs. Let's suppose that you mean "the body parts used for speech", but not necessarily in the way that they are used in speech. (Ingressive lung air uses the anatomy, but in a non-...


4

Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word prevalence for 61,800 English words by Marc Brysbaert, Paweł Mandera, Samantha McCormick, and Emmanuel Keuleers in the link given.


4

I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours vertical format: One token per line with annotations separated by TABs inline format: plain text with annotations appended to each token like this: plain/ADJ text/NOUN with/PREP ... (the seperating character ...


4

There are several promising new developments where JSON is being used to provide improved modern data formats for computational linguistics, NLP, and corpus research. There is an extensive project under way to create a JSON framework for linguistic fieldwork resources, including lexicon, texts, etc: Hieber, Daniel W. (2018). Data Format for Digital ...


4

Whoops, didn't know I was allowed to answer my own question. Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely proportional to its rank in frequency, so the most common word is used twice as much as the second most common word, three times as much as the third most common, and ...


4

For the purpose of normalisation, you should use a formula such as this: normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000 This formula can be adjusted to normalise, for example, per 100,000 words - just replace the 1,000,000 in the formula by 100,000. When calculating log likelihood for a comparison of the frequency of a ...


4

Good question. Constituency is the theory behind such a tree diagram. There are a bunch of different Constituency tests which you can do on paper: [ [John] [ [hit] [ [ [the] ] [ball] ] ] ] Wh-substitution: [John]: Who hit the ball? [the]: John hit which ball? Wh-substitution and do-support-substitution: [hit]: John did what to the ball? Importantly, ...


4

ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this. (The main reason people do that is because they're about to translate it into 100 languages, and would rather fix the errors just once.) There are two ways to try to do what you want to do. 1. Just pick a dummy target language, like English Let's ...


3

I think, the key point answering this question would be by understanding the distinction between an alphabet and a script. Some boring definitions Alphabet is defined as a set of letters. Examples are Latin, Cyrillic, etc. Script, or, more formally, Writing system, is a language-specific set of rules used to encode a text in a particular language. For ...


Only top voted, non community-wiki answers of a minimum length are eligible