A database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

learn more… | top users | synonyms (1)

1
vote
0answers
27 views

colloquial English corpus with time expressions

I'm looking for a colloquial English corpus that has time expressions marked in it. I don't need the time expressions to be coded in any special way, just that they be marked so I can find them. All ...
1
vote
1answer
21 views

Open corpora of personal names

The only open corpus of personal names of which I am aware is the Names Corpus by Mark Kantrowitz and Bill Ross (which can also be found in the Python Natural Language Toolkit (NLTK)). It is much ...
0
votes
0answers
22 views

Corpus with sentences translated to English

Does anyone know of a corpus I could use in my research?, for any language, where each sentence is translated correctly into English. In particular, the corpus should contain primarily everyday spoken ...
1
vote
0answers
18 views

Corpus which includes smileys or even emojies

for a little project in University my group would like to have a look at how emojies or smileys are used in order to support or express irony, sarcasm or similiar. Currently we are trying to find a ...
0
votes
1answer
21 views

Corpora Tools for Cataloguing and Phonetic Markup?

I'm creating an annotated corpus of poetry annotated with accent, alliteration, rhythm, and phonetics. I'd like it to handle and input poetry in multiple languages but still be able to add markup ...
2
votes
2answers
63 views

Looking for korean text corpus

I'm looking for a list of text corpus for korean. Is there any web site where I can find it? Regards.
3
votes
2answers
41 views

American English Corpus with spoken language exchange

I'm looking for an American English Corpus with spoken language exchange. There are many terms used. E.g. eTandem, tandem, etc. TIA
1
vote
0answers
18 views

Should I use the Log-Likelihood (G2) test to compare word frequency across corpora?

I need to compare the frequency of occurrence of a word in two corpora. Take for example the word fail which occurs 311 times in a corpus containing language from the sciences (2 million tokens) and ...
1
vote
0answers
30 views

Why doesn't GeneTag use BIO for its annotations?

GeneTag (1) is a tagged corpus for gene/protein named entity recognition. The corpus is annotated as follows: P00089778A0000 The_TAG concentration_TAG of_TAG alpha_GENE1 2_GENE1 -_GENE1 ...
1
vote
0answers
21 views

Political Advertising Corpus

Is there a large political advertising corpus available for download? I am working on a project but the Wesleyan and University of Wisconson corpus are off limits to me. I sent a request on official ...
0
votes
2answers
73 views

Do we have a taxonomy other than part of speech?

I'm trying to create an educational system, and I would like to have more taxonomies than a simple part of speech. For example, I would like to be able to categorize words into: Tree => object ...
2
votes
2answers
43 views

What are some canonical or seminal corpus linguistic studies using the Brown Corpus?

I'm looking for some good works of corpus linguistics using the Brown Corpus. Does anyone know of an important early work, and/or an important recent work that uses it?
2
votes
0answers
24 views

Slot filling corpora

I was wondering if anyone knows of slot filling corpora such as ATIS. I tried finding ATIS but since I am not a member of LDC I couldnt get access to it. Do you know of any place where I can find a ...
0
votes
0answers
22 views

Phrase suggestion model using a gold standard corpus

There are already spell checking models available which help us to find the suggested correct spellings based on a corpus of trained correct spellings. Can the granularity be increased to "word" from ...
1
vote
2answers
256 views

Most commonly used words across languages?

I'm doing my own (very small) corpus analysis, including a word frequency list and would like to compare the top, say, 30 words with other such lists. I know that there are things like the GSL/NGSL, ...
2
votes
2answers
144 views

Do standard corpus analysis tools like AntConc and Wordsmith work for foreign languages?

Can I use tools like AntConc and Wordsmith and produce decent results when analyzing languages like French, German or Spanish? Would it deal with special symbols like "à", "ü", "ß", "ç" and similar ...
2
votes
1answer
46 views

Question type corpus

I'm looking for a corpus that contains questions and question types. Like in this article, but they didn't attach any link to their corpus. http://polibits.gelbukh.com/2009_40/40_01.pdf
1
vote
0answers
42 views

Does corpus linguistics research confirm/disconfirm Bolinger principle?

I remember learning about Bolinger's principle (basically, when there is a verb complement situation like "I like [verb complement]" or "I want to [verb complement]", gerunds are preferred to express ...
1
vote
1answer
34 views

How do you search for nouns that contain the sequence of letters sb in the Corpus of Contemporary American English?

I am trying to design a word list for a phonetics project. I can use wildcard searches in the corpus such as * sb * which will return all words that contain this sequence of letters (either medially ...
0
votes
1answer
110 views

Special Characters and Uppercase Frequency?

Is there any way to have a good heuristic study on the frequency of special characters and uppercase ? Would studying a freely available corpus (around 15m words) generates a good estimation? Edit: ...
1
vote
1answer
217 views

Biggest freely available English corpus?

Any help on finding the biggest freely available English corpus that can be used on research? So far I have found OANC with 15 M words.
0
votes
1answer
43 views

Reason for using 4- and 5-grams for co-occurrence analysis

I am trying to replicate the results of a paper which didn't share its data methods that clearly. It makes use of the Web 1T 5-gram Version 1 data set collected from Google to expand 72 words into 964 ...
3
votes
0answers
24 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
2
votes
2answers
157 views

Conversational English corpus for download

I have a project which requires a corpus of conversational English in plain text (although I can perform some processing as needed). Since I am a student, I need to find a corpus that is free and ...
0
votes
0answers
38 views

Parallel dependency grammars

Is there something like ParGram but dependency-based? Note that I'm not looking for a corpus but for a formal grammar (set of rules) with a parser and a corpus of test sentences.
1
vote
0answers
32 views

Researching extraposition in a syntactic treebank

I'm writing a paper on extraposition in English (and other right-branching discontinuities). I have found a lot of interesting theory on this but the instructions say that if possible concrete corpus ...
1
vote
1answer
89 views

Evaluate idea to autobuild russian-english parallel corpus

I was looking for russian-english parallel corpus, but have found none publicly available to the moment. If you know one, or parallel text database, I would be very grateful for one. (I heard of ...
1
vote
3answers
125 views

Is it possible to build thesaurus automatically?

Is it possible to build thesaurus by simply analyzing lots of text, or it's obligatory should be handmade? Also a question: is there free-to-download&use English language thesaurus?
0
votes
2answers
49 views

Expected Frequency

I am reading an article where the authors mention something they call the expected frequency of a word in a corpus without explaining or giving a definition of what this means. The article in ...
3
votes
0answers
75 views

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Alternatively, do any larger corpora have categories that pertain to technology, computer science, etc.? Or collections of English documentation and/or manuals related to modern devices, interfaces, ...
1
vote
1answer
96 views

Word difficulty as a function of frequency

What is known in general about the relationship between word difficulty and word frequency? That is, for a given word, suppose I know the frequency with which it (or its word stem) is encountered in ...
5
votes
1answer
78 views

How should grammatical errors be treated in parsing and tokenisation?

I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are ...
0
votes
1answer
51 views

German sentence corpus with cloze norms

I wish to conduct research on German based on cloze probabilities. Cloze probabilities are not trivial to collect, so I am asking if there exist any publicly available corpora of German sentences with ...
6
votes
1answer
217 views

copyright issues when making a corpus from the web

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I ...
2
votes
1answer
120 views

Time annotated corpus: plain text english corpus

I am working on how entities take a new sense over time. I am trying to find out a large english corpus (free to download) which should have the time annotation of the origin of the text. I suppose ...
0
votes
0answers
68 views

How to annotate a corpus of written discourse for pragmatic analysis (speech acts)?

I am currently creating a small-ish corpus of online conversations that I wish to analyze quantitatively and qualitatively from a pragmatic perspective (speech acts level mainly). The problem is that ...
1
vote
0answers
59 views

Is there a data set of elementary typical phrases translated in different languages?

To be more explicit, I think about a collection of "abstract" sentences which could be categorized in such a way that they would be easily identified in any language. For example, some typical phrases ...
2
votes
1answer
81 views

What is considered the smallest possible sample size for word frequency lists used in FL instruction?

I've been engaged in a conversation on another site pertaining to frequency analysis, particularly in relation to the 1966 work Buntús Gaeilge, Colmán Ó Huallacháin; Ireland. Department of ...
1
vote
1answer
62 views

What are lexical and morpho-syntactic alternations?

I ought to write a paper with corpus-based analysis of a lexical or morpho-syntactic alternation. In other words, the paper should deal with two (or more) nearly synonymous lexical items or ...
0
votes
1answer
52 views

Switchboard queries with NXT (the TEXT function)

The NXT query language includes the query function TEXT, which should return the textual content of the variable it applies to. In theory, Switchboard should support this. The file ...
3
votes
2answers
134 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
2
votes
0answers
48 views

What are the subsets of the Eijiro corpus about?

I'm working with 2 japanese corpora called 'waei' and 'reiji'. They are legacy data, so I have no README or else to tell me the topic of either of them. But I need to know what they are about. My ...
3
votes
0answers
116 views

Is there any corpus for idioms?

I'm looking for a corpus for English (American, GB, Australian) idioms. Preferably created manually, because I already have two, but they are rather small and were built semi-automatically.
2
votes
1answer
206 views

Comparing frequencies of two corpora

I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
2
votes
1answer
78 views

Is there a web-based corpus tools that I can upload and use with my own corpus?

I am looking for a ready-made concordancer or collocator with a simple search interface that I can upload on my website and have my students use to search a custom corpus. I have been working with ...
2
votes
1answer
830 views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
1
vote
0answers
176 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
7
votes
3answers
2k views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
5
votes
1answer
210 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
0
votes
0answers
70 views

Tabulated lists of examples

"Everybody [?] knows" that there are these pairs of corresponding words in German and English in which an "f" appears in German where a "p" appears in English: Bischoff, bishop Schiff, ship ...