Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
1
vote
1answer
35 views

List of English words with usage samples

For my learning tool project I need a free English word list with the usage sample of each word and/or its definition. For example: /word/ Apple /phrase/ This apples are big and red. /definition/ ...
3
votes
0answers
55 views

Are there any publicly available spell checking corpora?

I need some corpora that contain sentences with misspelled word(s) in order to evaluate the performance of my own spell checking approach. So, the corpora should define the right word alongside the ...
1
vote
2answers
300 views

I am building a chatbot and I need corpora

I am building a chatbot and I need corpora to train them. I need corpora that is used as general chatting. I would be happy if anyone would help me with it.
5
votes
1answer
125 views

Parts-of-speech tagging and finding relevant phrases in documents

I've got a corpus of half a million text documents. I'd like to identify phrases in each document that are the most descriptive with which to build tag clouds. Let's say that I identify the most ...
2
votes
0answers
21 views

How to get data for ACADEMIC + SPOKEN (Academic speech) simultaneously using BNC?

I'm looking for filler words used in academia. Since fillers are mainly used in spoken discourse, I need to look for academic spoken 'texts'. Anyone knows how to do this using BNC? (Explanation: If ...
2
votes
0answers
182 views

Differences in collective nouns and agreement between Ame and BrE

I am trying to use two corpora,the BNC and COCA to ascertain the difference in instances of single and plural verb agreement between American and British English. I have some interesting ideas for ...
1
vote
2answers
90 views

Criminal Text Corpora

My goal is to bolster a database with charge type categories and decipher between, criminals, victims, interested parties, and suspects. I have searched for a corpora of text that may already have ...
1
vote
1answer
279 views

Corpus for anaphora resolution [closed]

I need a test and train corpus for my project containing anaphora. It should atleast have 100 sentences. please help asap?
1
vote
0answers
41 views

Middle and Old English Corpora Distributable Under Open Source License

I'm working on an open source computational linguistics project, and I need Old and Middle English corpora that I can distribute along with the software. Most of the corpora that I've been able to ...
0
votes
0answers
149 views

Corpus of tagged text (English newspapers or any tagged text)

I'm developing a system to extract tags from text (English) and currently I have no dataset to test the system and evaluate, could someone point me to a source (preferably a free one) thanks SO I ...
1
vote
1answer
23 views

Related to Corpa Query software

Some time ago, a colleague at an institution I used to work at gave a presentation to the faculty on using corpora queries. She used [I think a website, though it could have been a software] where you ...
0
votes
0answers
111 views

how may I identify informal synonyms and abbreviations in a corpus?

I am interested in atherogenesis which is the initiating process for atherosclerosis which in turn, can lead to ischaemic/ischemic stroke. Is there a recognised process/algorithm - ideally using a ...
0
votes
0answers
51 views

Significance Testing in corpus linguistics for corpus comparison

I would like to compare two corpora, one consisting of Wikipedia articles and one made up of articles in printed encyclopedias. The corpus is thematically comparable because I always chose the same ...
1
vote
0answers
192 views

colloquial English corpus with time expressions

I'm looking for a colloquial English corpus that has time expressions marked in it. I don't need the time expressions to be coded in any special way, just that they be marked so I can find them. All ...
2
votes
1answer
426 views

Open corpora of personal names

The only open corpus of personal names of which I am aware is the Names Corpus by Mark Kantrowitz and Bill Ross (which can also be found in the Python Natural Language Toolkit (NLTK)). It is much ...
0
votes
0answers
57 views

Corpus with sentences translated to English

Does anyone know of a corpus I could use in my research?, for any language, where each sentence is translated correctly into English. In particular, the corpus should contain primarily everyday spoken ...
1
vote
0answers
52 views

Corpus which includes smileys or even emojies

for a little project in University my group would like to have a look at how emojies or smileys are used in order to support or express irony, sarcasm or similiar. Currently we are trying to find a ...
1
vote
2answers
88 views

Corpora Tools for Cataloguing and Phonetic Markup?

I'm creating an annotated corpus of poetry annotated with accent, alliteration, rhythm, and phonetics. I'd like it to handle and input poetry in multiple languages but still be able to add markup (...
5
votes
3answers
2k views

Looking for korean text corpus

I'm looking for a list of text corpus for korean. Is there any web site where I can find it? Regards.
5
votes
3answers
202 views

American English Corpus with spoken language exchange

I'm looking for an American English Corpus with spoken language exchange. There are many terms used. E.g. eTandem, tandem, etc. TIA
2
votes
0answers
91 views

Should I use the Log-Likelihood (G2) test to compare word frequency across corpora?

I need to compare the frequency of occurrence of a word in two corpora. Take for example the word fail which occurs 311 times in a corpus containing language from the sciences (2 million tokens) and ...
0
votes
0answers
68 views

Why doesn't GeneTag use BIO for its annotations?

GeneTag (1) is a tagged corpus for gene/protein named entity recognition. The corpus is annotated as follows: P00089778A0000 The_TAG concentration_TAG of_TAG alpha_GENE1 2_GENE1 -_GENE1 ...
1
vote
0answers
63 views

Political Advertising Corpus

Is there a large political advertising corpus available for download? I am working on a project but the Wesleyan and University of Wisconson corpus are off limits to me. I sent a request on official ...
0
votes
2answers
301 views

Do we have a taxonomy other than part of speech?

I'm trying to create an educational system, and I would like to have more taxonomies than a simple part of speech. For example, I would like to be able to categorize words into: Tree => object ...
2
votes
2answers
83 views

What are some canonical or seminal corpus linguistic studies using the Brown Corpus?

I'm looking for some good works of corpus linguistics using the Brown Corpus. Does anyone know of an important early work, and/or an important recent work that uses it?
3
votes
1answer
120 views

Slot filling corpora

I was wondering if anyone knows of slot filling corpora such as ATIS. I tried finding ATIS but since I am not a member of LDC I couldnt get access to it. Do you know of any place where I can find a ...
0
votes
0answers
36 views

Phrase suggestion model using a gold standard corpus

There are already spell checking models available which help us to find the suggested correct spellings based on a corpus of trained correct spellings. Can the granularity be increased to "word" from ...
1
vote
2answers
3k views

Most commonly used words across languages?

I'm doing my own (very small) corpus analysis, including a word frequency list and would like to compare the top, say, 30 words with other such lists. I know that there are things like the GSL/NGSL, ...
2
votes
2answers
761 views

Do standard corpus analysis tools like AntConc and Wordsmith work for foreign languages?

Can I use tools like AntConc and Wordsmith and produce decent results when analyzing languages like French, German or Spanish? Would it deal with special symbols like "à", "ü", "ß", "ç" and similar ...
4
votes
1answer
648 views

Question type corpus

I'm looking for a corpus that contains questions and question types. Like in this article, but they didn't attach any link to their corpus. http://polibits.gelbukh.com/2009_40/40_01.pdf
1
vote
0answers
82 views

Does corpus linguistics research confirm/disconfirm Bolinger principle?

I remember learning about Bolinger's principle (basically, when there is a verb complement situation like "I like [verb complement]" or "I want to [verb complement]", gerunds are preferred to express ...
1
vote
1answer
65 views

How do you search for nouns that contain the sequence of letters sb in the Corpus of Contemporary American English?

I am trying to design a word list for a phonetics project. I can use wildcard searches in the corpus such as * sb * which will return all words that contain this sequence of letters (either medially ...
0
votes
1answer
628 views

Special Characters and Uppercase Frequency?

Is there any way to have a good heuristic study on the frequency of special characters and uppercase ? Would studying a freely available corpus (around 15m words) generates a good estimation? Edit: ...
8
votes
7answers
2k views

Biggest freely available English corpus?

Any help on finding the biggest freely available English corpus that can be used on research? So far I have found OANC with 15 M words.
0
votes
1answer
108 views

Reason for using 4- and 5-grams for co-occurrence analysis

I am trying to replicate the results of a paper which didn't share its data methods that clearly. It makes use of the Web 1T 5-gram Version 1 data set collected from Google to expand 72 words into 964 ...
3
votes
1answer
317 views

Different discounting methods with SRILM toolikt

I want to do some smoothing on a corpus and compare the results. for train language model without smoothing I used "-addsmooth 0" option now I should add two different smoothing and compare them, I ...
3
votes
3answers
2k views

Conversational English corpus for download

I have a project which requires a corpus of conversational English in plain text (although I can perform some processing as needed). Since I am a student, I need to find a corpus that is free and ...
1
vote
0answers
44 views

Researching extraposition in a syntactic treebank

I'm writing a paper on extraposition in English (and other right-branching discontinuities). I have found a lot of interesting theory on this but the instructions say that if possible concrete corpus ...
1
vote
1answer
265 views

Evaluate idea to autobuild russian-english parallel corpus

I was looking for russian-english parallel corpus, but have found none publicly available to the moment. If you know one, or parallel text database, I would be very grateful for one. (I heard of ...
1
vote
4answers
203 views

Is it possible to build thesaurus automatically?

Is it possible to build thesaurus by simply analyzing lots of text, or it's obligatory should be handmade? Also a question: is there free-to-download&use English language thesaurus?
0
votes
2answers
129 views

Expected Frequency

I am reading an article where the authors mention something they call the expected frequency of a word in a corpus without explaining or giving a definition of what this means. The article in ...
7
votes
2answers
892 views

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Alternatively, do any larger corpora have categories that pertain to technology, computer science, etc.? Or collections of English documentation and/or manuals related to modern devices, interfaces, ...
2
votes
1answer
341 views

How to determine difficulty of a word if its frequency in a corpus is known?

What is known in general about the relationship between word difficulty and word frequency? That is, for a given word, suppose I know the frequency with which it (or its word stem) is encountered in ...
5
votes
1answer
101 views

How should grammatical errors be treated in parsing and tokenisation?

I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are ...
0
votes
1answer
189 views

German sentence corpus with cloze norms

I wish to conduct research on German based on cloze probabilities. Cloze probabilities are not trivial to collect, so I am asking if there exist any publicly available corpora of German sentences with ...
7
votes
2answers
924 views

Do I have copyright issues when making a corpus from the web?

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I ...
4
votes
2answers
258 views

Time annotated corpus: plain text english corpus

I am working on how entities take a new sense over time. I am trying to find out a large english corpus (free to download) which should have the time annotation of the origin of the text. I suppose ...
0
votes
0answers
132 views

How to annotate a corpus of written discourse for pragmatic analysis (speech acts)?

I am currently creating a small-ish corpus of online conversations that I wish to analyze quantitatively and qualitatively from a pragmatic perspective (speech acts level mainly). The problem is that ...
1
vote
0answers
62 views

Is there a data set of elementary typical phrases translated in different languages?

To be more explicit, I think about a collection of "abstract" sentences which could be categorized in such a way that they would be easily identified in any language. For example, some typical phrases ...
2
votes
1answer
150 views

What is considered the smallest possible sample size for word frequency lists used in FL instruction?

I've been engaged in a conversation on another site pertaining to frequency analysis, particularly in relation to the 1966 work Buntús Gaeilge, Colmán Ó Huallacháin; Ireland. Department of ...