Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
18
votes
1answer
8k views

Does anyone know of text message corpora?

I am looking for a large corpus of text messages. By large, I am hoping to have at least 15,000 text messages in my sample. I am fine with combining several smaller corpora into a larger corpus as I ...
1
vote
0answers
21 views

corpus search in Sketch Engine for particular correlative coordinators

Can someone teach me/show me how to search for either...or, neither...or/nor, whether...or and etc. in Sketch engine (in BAWE corpus particularly)? I have read basically every guide and watched a lot ...
4
votes
2answers
289 views

Is there a corpus of Arabic text that doesn't include quotations of the Quran?

I want to search such a corpus to determine the commonality or in other terms the frequency of use of the word Quran القرآن in comparison to The Generous Quran القرآن الكريم. But the condition is that ...
1
vote
0answers
35 views

Error of word frequency [closed]

I have created a database of lyrics with some 10000 songs and did a word frequency analysis to find the most common words. Next, i sorted the corpus according to the publication date and measured the ...
2
votes
1answer
68 views

Are there any computer readable corpora for ASL?

I'm looking to see if there are any free-to-use corpora in ASL (any domain is fine), transcribed so a computer can use it. I am interested in doing some analysis on ASL syntax, morphology etc. But I ...
3
votes
1answer
109 views

Classifying the verbs in a small corpus

I have a relatively small language corpus (ca. 250000 tokens). There are two possible stems for a verb in the language: active and passive. Some of the verbs in the language occur only in passive ...
9
votes
1answer
9k views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
2
votes
1answer
235 views

building a corpus from COCA KWIC

I'd like to look at an adjective + clause (adverbial, to-infinitival, -ing) combination in COCA and I'm happy with the KWIC results I got so far. For my further analysis, it would be useful to have ...
1
vote
2answers
387 views

Dictionary building / corpus exploring software

What software do lexicographers use these days to manage and search corpora? Is there a standard solution used by most people, or does every dictionary commission their own software to suit their own ...
0
votes
0answers
92 views

Corpus for text classification with annotated support phrases

I am looking for corpora for text classification with annotated support phrases. Support phrases = phrases that explain why a text was given a label. For example, if the text is "The patient works ...
2
votes
1answer
180 views

Open corpora for modern English

I need a corpus of modern English, with part-of-speech tags, in order to train language models, specifically a part-of-speech tagger. The domain I am doing this in is the Hansard, the transcript of ...
1
vote
0answers
140 views

Synonym/hyponyms/hypernym detection data set

I am looking for data sets that I could use to train and benchmark a system that detects synonyms, hyponyms and hypernyms in a text. For example, in the sentence In statistics, logistic ...
1
vote
1answer
37 views

How to deal with “it's” when using Miller Newman Friedman list in a corpus?

I'm using their list for a project that incorporates some techniques of authorship analysis. I don't know what to do with "it's", though. I obviously cannot ignore the apostrophe and count it as "...
3
votes
2answers
79 views

Comparative Methodology

Firstly many apologies if this question is not appropriate for this forum; I am a mathematician and a mathematics education researcher. However, I am posting on this forum because I am doing research ...
0
votes
0answers
102 views

Library of regular expressions for lexical relations in a corpus

I would like to ask if there is a library of regular expressions for mining patterns reflecting lexical semantic relations from a corpus. These patterns should cover a wide variety of aspects synonymy ...
1
vote
1answer
363 views

Corpus Linguistics: How do I compare date from two corpora correctly?

I want to count the occurrences of the English “wanna” structure diachronically, in order to create a frequency per century chart. My first step was to use the Corpus of Historical American English (...
1
vote
0answers
18 views

How to (automatically ) detect the steady-part of vowels?

Please I need your help ASAP, I need to segment an audio corpus of 4 hours , and I need a tool or script to detect the steady part of vowels and place the boundaries .
3
votes
0answers
158 views

Distribution of the set of meanings of a given word, in a corpus

I am looking to know about the distribution of word senses, of a particular word in a corpus. How often a word sense will occur out of all the occurrences of that word with any meaning. For example: ...
2
votes
1answer
295 views

Phonetics - English Pronunciation of Vowels Corpus

I am a physics student so forgive me if I'm not very precise with the language. I am currently taking an English course, and my teacher is specialized in teaching pronunciation. I am interested in ...
0
votes
1answer
179 views

Is there a resource for English relational nouns

Relational nouns, like brother, mother, capital, advisor, leader are different from sortal words like cup, person, or woman. They differ in the sense that, one cannot decide if x is a brother unto ...
0
votes
1answer
199 views

How to determine who is the author of a message in the nps_chat corpora?

I try to use nps_chat corpora provided by nltk library. I follow the example given here. In particular, I can reproduce this results: >>> from nltk.corpus import nps_chat >>> ...
0
votes
1answer
73 views

word relatedness, defining character of a word

How can NLP tell the word "stool" is closely related to the word "dirty?" When the word relationship is based on a corpus, most likely the word "stool" is related to some medical terms such as "bloody"...
6
votes
2answers
3k views

How to establish word frequencies from a corpus?

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
7
votes
2answers
928 views

Do I have copyright issues when making a corpus from the web?

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I ...
2
votes
1answer
343 views

How to determine difficulty of a word if its frequency in a corpus is known?

What is known in general about the relationship between word difficulty and word frequency? That is, for a given word, suppose I know the frequency with which it (or its word stem) is encountered in ...
1
vote
2answers
819 views

French corpus with frequency list of POS tagged words (not lemmas)

I'm trying to find a corpus (even purchase it) of French language that has these characteristics: It should has a frequency list List of words (not just lemmas) which are POS tagged Preferably taken ...
3
votes
0answers
55 views

Are there any publicly available spell checking corpora?

I need some corpora that contain sentences with misspelled word(s) in order to evaluate the performance of my own spell checking approach. So, the corpora should define the right word alongside the ...
1
vote
1answer
35 views

List of English words with usage samples

For my learning tool project I need a free English word list with the usage sample of each word and/or its definition. For example: /word/ Apple /phrase/ This apples are big and red. /definition/ ...
1
vote
2answers
88 views

Corpora Tools for Cataloguing and Phonetic Markup?

I'm creating an annotated corpus of poetry annotated with accent, alliteration, rhythm, and phonetics. I'd like it to handle and input poetry in multiple languages but still be able to add markup (...
5
votes
1answer
125 views

Parts-of-speech tagging and finding relevant phrases in documents

I've got a corpus of half a million text documents. I'd like to identify phrases in each document that are the most descriptive with which to build tag clouds. Let's say that I identify the most ...
1
vote
2answers
301 views

I am building a chatbot and I need corpora

I am building a chatbot and I need corpora to train them. I need corpora that is used as general chatting. I would be happy if anyone would help me with it.
2
votes
0answers
21 views

How to get data for ACADEMIC + SPOKEN (Academic speech) simultaneously using BNC?

I'm looking for filler words used in academia. Since fillers are mainly used in spoken discourse, I need to look for academic spoken 'texts'. Anyone knows how to do this using BNC? (Explanation: If ...
0
votes
0answers
149 views

Corpus of tagged text (English newspapers or any tagged text)

I'm developing a system to extract tags from text (English) and currently I have no dataset to test the system and evaluate, could someone point me to a source (preferably a free one) thanks SO I ...
2
votes
0answers
182 views

Differences in collective nouns and agreement between Ame and BrE

I am trying to use two corpora,the BNC and COCA to ascertain the difference in instances of single and plural verb agreement between American and British English. I have some interesting ideas for ...
1
vote
2answers
90 views

Criminal Text Corpora

My goal is to bolster a database with charge type categories and decipher between, criminals, victims, interested parties, and suspects. I have searched for a corpora of text that may already have ...
1
vote
1answer
279 views

Corpus for anaphora resolution [closed]

I need a test and train corpus for my project containing anaphora. It should atleast have 100 sentences. please help asap?
1
vote
0answers
41 views

Middle and Old English Corpora Distributable Under Open Source License

I'm working on an open source computational linguistics project, and I need Old and Middle English corpora that I can distribute along with the software. Most of the corpora that I've been able to ...
0
votes
1answer
23 views

Related to Corpa Query software

Some time ago, a colleague at an institution I used to work at gave a presentation to the faculty on using corpora queries. She used [I think a website, though it could have been a software] where you ...
0
votes
0answers
111 views

how may I identify informal synonyms and abbreviations in a corpus?

I am interested in atherogenesis which is the initiating process for atherosclerosis which in turn, can lead to ischaemic/ischemic stroke. Is there a recognised process/algorithm - ideally using a ...
2
votes
2answers
762 views

Do standard corpus analysis tools like AntConc and Wordsmith work for foreign languages?

Can I use tools like AntConc and Wordsmith and produce decent results when analyzing languages like French, German or Spanish? Would it deal with special symbols like "à", "ü", "ß", "ç" and similar ...
0
votes
0answers
51 views

Significance Testing in corpus linguistics for corpus comparison

I would like to compare two corpora, one consisting of Wikipedia articles and one made up of articles in printed encyclopedias. The corpus is thematically comparable because I always chose the same ...
0
votes
1answer
190 views

German sentence corpus with cloze norms

I wish to conduct research on German based on cloze probabilities. Cloze probabilities are not trivial to collect, so I am asking if there exist any publicly available corpora of German sentences with ...
1
vote
0answers
192 views

colloquial English corpus with time expressions

I'm looking for a colloquial English corpus that has time expressions marked in it. I don't need the time expressions to be coded in any special way, just that they be marked so I can find them. All ...
1
vote
1answer
429 views

Open corpora of personal names

The only open corpus of personal names of which I am aware is the Names Corpus by Mark Kantrowitz and Bill Ross (which can also be found in the Python Natural Language Toolkit (NLTK)). It is much ...
0
votes
0answers
57 views

Corpus with sentences translated to English

Does anyone know of a corpus I could use in my research?, for any language, where each sentence is translated correctly into English. In particular, the corpus should contain primarily everyday spoken ...
1
vote
0answers
52 views

Corpus which includes smileys or even emojies

for a little project in University my group would like to have a look at how emojies or smileys are used in order to support or express irony, sarcasm or similiar. Currently we are trying to find a ...
1
vote
0answers
91 views

Should I use the Log-Likelihood (G2) test to compare word frequency across corpora?

I need to compare the frequency of occurrence of a word in two corpora. Take for example the word fail which occurs 311 times in a corpus containing language from the sciences (2 million tokens) and ...
0
votes
0answers
68 views

Why doesn't GeneTag use BIO for its annotations?

GeneTag (1) is a tagged corpus for gene/protein named entity recognition. The corpus is annotated as follows: P00089778A0000 The_TAG concentration_TAG of_TAG alpha_GENE1 2_GENE1 -_GENE1 ...
0
votes
1answer
630 views

Special Characters and Uppercase Frequency?

Is there any way to have a good heuristic study on the frequency of special characters and uppercase ? Would studying a freely available corpus (around 15m words) generates a good estimation? Edit: ...
1
vote
0answers
63 views

Political Advertising Corpus

Is there a large political advertising corpus available for download? I am working on a project but the Wesleyan and University of Wisconson corpus are off limits to me. I sent a request on official ...