22
Corpora containing more than 15 million words are often not freely available due to copyright issues (such as the British National Corpus and the Corpus of Contemporary American English).
The open part of the American National Corpus (OANC) might fulfill your criteria. It contains almost 15 m. words, it's free, and contains conversations and other genres. ...
12
Here is a variety of language corpora with millions of sentences each: http://corpora.informatik.uni-leipzig.de/download.html
11
This is a complex topic but here's an attempt at an answer. My background is in describing and documenting relatively small, endangered languages, so I'll describe how it's done in that situation. I'll assume you have computers, software, internet, native speaking informants, recording equipment, and many years to spend on the work.
Firstly, recorded texts ...
9
As with all natural laws, Zipf's law is an approximation. If you take a large corpus, and compute the Zipf curve, it will more or less follow a Zipf distribution (with coefficients thrown in to account for slack).
This doesn't mean that for every language it follows the exact rule of 'the second most common lexical item is 1/2 as frequent as the most common'...
9
In addition to the NPS chat corpus included with NLTK, also see NUS text message corpus for text conversations. You can also use the Twitter API to build a corpus of tweets as well.
8
Zipf’s law, as I understand it, is not really about languages, but about statistics and probability. It is just one of several formulations of the fact that many non-arbitrary sequences of numbers (frequency of words in a given corpus; population size of cites in relation to their rank; annual turnover of ranked companies; etc., etc.) are not evenly ...
8
There are many spoken English corpora available. But generally, you need to ask more questions than 'plain text' before you find the right one. Length, level of annotation, format of annotation, type of conversation, genre/register, dialect, natural vs. elicited, etc. Those will all depend on the type of research questions you want to answer.
If you just ...
8
For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical errors automatically.
We improved on that for the Royal Society Corpus and our scripts to do that are available for download here. Our approach is tuned for ...
7
I can only speak for Germany, and IANAL (I am not a lawyer). The situation is basically as follows:
You can collect material from accessible sources (from the web, from radio broadcasts, from TV) and do analyses on that material
You can do so within a closed collaboration with some collaborators including students and guests visiting your institution (they ...
answered Jul 5 '19 at 13:15
jk - Reinstate Monica
22.7k3 gold badges53 silver badges97 bronze badges
6
The Grupo de Ingeniería Lingüística, part of the Universidad Nacional Autónoma de México has a WhatsApp corpus of undergraduate students. They have a paper that introduces it: http://www.aclweb.org/anthology/W18-3501.
They have their own web-based corpus management tool http://www.corpus.unam.mx/geco/ but I don't see the WhatsApp corpus on there, so you ...
5
I looked at the SO discussion (and admit that I can't compute the consequences of all of the code). Those guys even admit that hyphenation solutions can't handle quasi-novel data (such as names of Welsh origin). Linguists can contribute linguistic clarity to the discussion.
I maintain that the SO answers do not yield correct syllabification, since ...
5
[ If I understand, you ideally want all meaningful phrases even where the head is not a noun, eg "save the day", "ready for action", "fantastically" or "supercalifragilisticexpialidocious". ]
You must break down the problem.
0. Sentence Segmentation and Tokenisation
I'm assuming this is already done.
1. Find named entities
You need a first pass to find ...
5
Two resources are particularly popular.
WordNet
Wiktionary
5
A straightforward rewriting of the Wikipedia formula gives
log V_R(n) = log K*n^beta
= log K + log n^beta
= log K + beta*log n
This allows us to identify K=C and beta=-alpha (probably the WSJ uses a different formulation of Heaps' law
V_R (n) = \frac{K}{n^\alpha}
). The remaining b is a strange additional parameter not present in ...
answered Apr 9 '19 at 15:16
jk - Reinstate Monica
22.7k3 gold badges53 silver badges97 bronze badges
5
Learner corpus
It seems that you're looking for what's commonly called "learner corpus", i.e. data that's written by people of various skill who don't speak the target language natively but are at various proficiency levels of learning it.
There are many such resources for English, the largest of which seems to be the Cambridge Learner Corpus, however, it ...
4
There are 3 steps in doing this task:
Identifying the languages in webpages
Building a crawler that downloads the web page
Doing linguistics analysis
since my primary NLP language is python and there are a lot of NLP libraries written for Python we use Python here.
For identifying the language you can use some great language identifiers like this (based ...
4
In English, you're fine with apostrophes. Compounds with hyphens, such as ice-cream, are mostly treated as separated tokens in corpora.
As for other languages in Europe, the "long" L in Catalan is denoted by a "bullet", e.g. pel·lícula.
4
The IMS Open Corpus Workbench might work for you.
4
Ironically, your best bet is: https://archive.org/download/stackexchange
4
Sketch Engine, a corpus manager and text analysis software, provides a few corpora with open access for research on https://app.sketchengine.eu/#open The largest English corpus (freely available) is ACL Anthology Reference Corpus with 62 million words.
On the other hand, you can try 30-day free trial of Sketch Engine and search one of the biggest English ...
4
You may find some useful ideas in the following list:
burp
Also belch. To expel gas noisily from the stomach through the mouth
The baby burped after being fed.
cough
To push air from the lungs in a quick, noisy explosion.
He started to cough once he had a cold .
hum
To make a sound from the vocal chords without ...
4
First, there are few sounds that people can make with only their mouths (to be more precise, the oral cavity). Clicks, yes, but not breathing, because breathing also involves the lungs. Let's suppose that you mean "the body parts used for speech", but not necessarily in the way that they are used in speech. (Ingressive lung air uses the anatomy, but in a non-...
4
Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word prevalence for 61,800 English words by Marc Brysbaert, Paweł Mandera, Samantha McCormick, and Emmanuel Keuleers in the link given.
answered Jun 21 '18 at 13:23
jk - Reinstate Monica
22.7k3 gold badges53 silver badges97 bronze badges
4
I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours
vertical format: One token per line with annotations separated by TABs
inline format: plain text with annotations appended to each token like this: plain/ADJ text/NOUN with/PREP ... (the seperating character ...
answered Jul 2 '18 at 10:31
jk - Reinstate Monica
22.7k3 gold badges53 silver badges97 bronze badges
4
There are several promising new developments where JSON is being used to provide improved modern data formats for computational linguistics, NLP, and corpus research.
There is an extensive project under way to create a JSON framework for linguistic fieldwork resources, including lexicon, texts, etc:
Hieber, Daniel W. (2018). Data Format for Digital ...
4
Whoops, didn't know I was allowed to answer my own question.
Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely proportional to its rank in frequency, so the most common word is used twice as much as the second most common word, three times as much as the third most common, and ...
4
For the purpose of normalisation, you should use a formula such as this:
normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000
This formula can be adjusted to normalise, for example, per 100,000 words - just replace the 1,000,000 in the formula by 100,000.
When calculating log likelihood for a comparison of the frequency of a ...
4
Good question. Constituency is the theory behind such a tree diagram. There are a bunch of different Constituency tests which you can do on paper:
[ [John] [ [hit] [ [ [the] ] [ball] ] ] ]
Wh-substitution:
[John]: Who hit the ball?
[the]: John hit which ball?
Wh-substitution and do-support-substitution:
[hit]: John did what to the ball?
Importantly, ...
4
ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this.
(The main reason people do that is because they're about to translate it into 100 languages, and would rather fix the errors just once.)
There are two ways to try to do what you want to do.
1. Just pick a dummy target language, like English
Let's ...
3
I think, the key point answering this question would be by understanding the distinction between an alphabet and a script.
Some boring definitions
Alphabet is defined as a set of letters. Examples are Latin, Cyrillic, etc.
Script, or, more formally, Writing system, is a language-specific set of rules used to encode a text in a particular language.
For ...
Only top voted, non community-wiki answers of a minimum length are eligible