Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
4
votes
0answers
52 views

PoS tagger - technical corpus

I try to find corpus with technical language to train my model, but I cannot find it. Do you know if there are any tagged PoS corpus in English related with technical language?
29
votes
5answers
1k views

Lists of linguistic resources

In the interest of cultivating a professional, academic community, I posted this question on Meta. One comment was to open a community-wiki question inviting others to contribute to a list of academic ...
3
votes
3answers
2k views

Conversational English corpus for download

I have a project which requires a corpus of conversational English in plain text (although I can perform some processing as needed). Since I am a student, I need to find a corpus that is free and ...
6
votes
1answer
405 views

How can I know if a log-likelihood score is high enough?

I am trying to understand if some terms can be considered as distinctly American or British. So for example I was studying the term "aubergine" and the normalized frequency in the American corpus is ...
3
votes
0answers
372 views

What is the formula for Usage Rate?

I read about a concept called the "usage rate" (proposed by Juliand and Chang-Rodriguez). It's a method for calculating the frequency of a word in a corpus. What exactly is the "usage rate" and what ...
5
votes
1answer
100 views

Word length as a function of word frequency?

I was wondering if any studies have been done in corpus linguistics which look at word length (number of phonemes per word) as a function of the word's frequency of use? Thanks! EDIT: Never mind, ...
1
vote
0answers
44 views

How to convert CHAT to TextGrid in a batch?

Siarad corpus is a corpus of spoken Welsh (~40 hours, all transcribed, glossed and translated!). I want to analyze some conversations using Praat. Praat uses the TextGrid format, but the Siarad corpus ...
2
votes
1answer
204 views

What is the state of the art for diacritic restoration?

Diacritic restoration is a term for guessing the original form of a text, after it's been converted to ASCII in a lossy way. In Lingála, for example, writing in ASCII merges two pairs of vowels and ...
13
votes
3answers
518 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
4
votes
1answer
45 views

Looking for Spanish email or chat corpora

I am looking for some corpora containing emails or chats in Spanish.
1
vote
3answers
90 views

Words with the hightest frequency that are not Collocates of each other (inside a text corpora)

Is there a way to obtain a: List, that has 2 words per row Each 2 words are not collocates (or collocations) of each other (ie: are not found within the same phrase inside the entire text corpus, or ...
1
vote
1answer
123 views

Dataset of subject-verb-object/adverb-verb/adjective-noun pairs

Does there exist a dataset which contains a list of subject-verb-object triplets or adverb-verb/adjective-noun pairs that occur within a certain (large) corpus? If not, what would be the most ...
2
votes
1answer
67 views

How to extract example sentences from linguistics papers

Does anyone know of a good tool to grab all example sentences used in a paper so they could be processed via an NLP pipeline? I have udpipe in mind especially: I'd like to take sentences from one or ...
4
votes
2answers
167 views

When analyzing a set of corpora, are there any standard practices with regard to the classification of gerunds?

In the article, "How Many Words Do You Need to Know in Spanish (or any other foreign language)? And WHICH Words Should You Be Learning?" I came upon the following: “Assume that a language learner ...
4
votes
1answer
191 views

Where can I find training data for dialects of Hindi?

I am working on a NLP project that aims at identifying different dialects of Hindi language. My über goal is to generate dialogs in once the program identifies which dialect is given. I tried ...
1
vote
2answers
283 views

Corpus for simple or basic english

I'm trying to find a good, extensive corpus that uses "simple" or "basic" English, in the sense that it should be easily understandable to people who are not proficient in the English language. For ...
3
votes
2answers
132 views

Which free resource to find word frequency?

My hypothesis is that math jargon is more aliening than other academic fields, and I want to concrete that by an example of the average frequency of the word in these two definitions of "...
2
votes
1answer
43 views

When corpus data suggests a word is common in a language?

I have two questions. If I were to use diachronic corpora to investigate the frequency of usage  for  words within  a word class of English that is mostly archaic today, how do I determine if the ...
0
votes
3answers
140 views

Who is corpus annotator

I'm a beginner of nlp. I'm wondering who is annotator of corpus usually ? Who does actually do works of annotating ? Researchers?
6
votes
3answers
2k views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
1
vote
1answer
41 views

What statistical test is appropriate in a corpus analysis of pronoun resolution preferences?

I have extracted data from a corpus that collected sentences with an antecedent and a next sentence with a pronoun or a stressed pronoun in subject position. Now, the anaphora can either refer back to ...
6
votes
0answers
71 views

Is there a threshold of usage of a word, below which it will become an archaism?

University student, beginning to learn about corpora, developing a research question for an assignment. There may well be linguistic concepts and terminology that already cover this idea and that I am ...
1
vote
0answers
129 views

Mandarin Chinese and Antconc

Can Antconc be used to build a corpus for Mandarin, Chinese? Specifically, could characters be recognized, or if one were to upload a pinyin text, could tones be displayed? Is this possible?
1
vote
1answer
299 views

Gold standard corpus size?

I have a 25 MB corpus (2.300.000 words) to annotate with a rule based system. What is the ideal size of a gold standard corpus that I should create in order to evaluate the tool? Many thanks for your ...
0
votes
0answers
28 views

How to determine word frequency based on demographic? (Does a public library of word frequencies exist by demographic)

I'm not sure "sign post" questions are appropriate however that is what this is. I am looking for a tool, a means of answering questions such as "Word X is used more by demographic Y than demographic ...
3
votes
0answers
64 views

Is there a frequency-ranked corpus of Punjabi NAV lemmata?

I am looking for a list of lexical roots for Punjabi, with frequency information whereby one can get an idea what the 1000 most-frequent words of the language are. I need to exclude case forms etc. ...
2
votes
2answers
143 views

glossary/dictionary corpus for NLP task

I am looking for glossary/dictionary corpus in machine readable form. Basically huge file/db which would contain word and its definition. Here is sample: Authenticity: a feature that characterizes ...
3
votes
1answer
366 views

Phonetically annotated speech corpus

Are there any phonetically annotated corpora of accented English speech? Preferably English spoken by native English speakers with a strong accents, such as speakers from a specific region in the UK ...
3
votes
2answers
258 views

Time annotated corpus: plain text english corpus

I am working on how entities take a new sense over time. I am trying to find out a large english corpus (free to download) which should have the time annotation of the origin of the text. I suppose ...
4
votes
3answers
2k views

Looking for korean text corpus

I'm looking for a list of text corpus for korean. Is there any web site where I can find it? Regards.
4
votes
3answers
202 views

American English Corpus with spoken language exchange

I'm looking for an American English Corpus with spoken language exchange. There are many terms used. E.g. eTandem, tandem, etc. TIA
14
votes
4answers
5k views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
4
votes
2answers
92 views

Tools and algorithms for finding non-verbatim quotations

I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite ...
2
votes
1answer
65 views

where can I find free corpus of spoken disabled people (in english, italian)

I am working on the research project in NLP domain. Basically, we are looking for a free audio corpus of spoken disabled people to study (for instance, aged people, or people has difficulty in normal ...
4
votes
3answers
2k views

Where can I find a free text corpus for the Hindi language?

I need a text corpus for my NLP project in Hindi, however all the corpora I found are not free for downloading. Is there any free annotated Hindi text corpus at all? If yes, where can I find it?
1
vote
0answers
25 views

Online data giving numbers and frequencies of nouns in some well-known novels?

Are there any freely available online data giving the numbers and frequencies of nouns used in some well-known novels, whether or not the novels are in the public domain? I mean data such as Dan ...
1
vote
1answer
433 views

Searching the word occurance in coca (corpus of contemporary American English) including meaning

I searched for a word inviting in coca and I got about 50 thousand usage examples. That's a lot of data. I'm interested in, if there is some possibility to refine the search, especially to search ...
0
votes
0answers
41 views

Corpus Assisted Discourse Studies

I wanted a corpus of recent English fiction (20 century onwards) which I can use as a reference corpus for my research work. It should consist of at least 5 million words. I do not need it in ...
1
vote
0answers
52 views

Corpora for sentence compression

I am looking for corpora for sentence compression. I am mostly interested in English, and any type of sentence. I am aware of: https://github.com/google-research-datasets/sentence-compression ...
1
vote
1answer
31 views

Corpora for extractive summaries

I am looking for corpora of extractive summaries. I am mostly interested in English, and any type of document.
0
votes
1answer
6k views

What is the difference between type and token?

My understanding of this is quite vague. Token I understand to be the total number of words in a given text, but type I am not so sure about. If I have a variety of inflected forms (e.g eat, eats, ...
3
votes
1answer
120 views

Slot filling corpora

I was wondering if anyone knows of slot filling corpora such as ATIS. I tried finding ATIS but since I am not a member of LDC I couldnt get access to it. Do you know of any place where I can find a ...
5
votes
2answers
770 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
2
votes
2answers
92 views

Command line tools for querying corpuses

I've built myself a little corpus that I'm using for language-learning. However, I'm reaching the limits of standard command-line tool-chain: grep and friends. For example, I might look for ...
20
votes
4answers
2k views

Corpus of baby-talk or motherese

Baby-talk or motherese is the language that parents tend to use when addressing preverbal or just-starting-to-speak children. What are the fundamental features or rules that define this subset of a ...
1
vote
1answer
42 views

Software for exploring large set of textual data

I am working with large amounts of text and already have a bunch of programmatic instruments to explore them (Python, R, etc). To say the truth, most of the exploration tasks like bigram extraction, ...
5
votes
2answers
329 views

Corpus Linguistics: Is it possible to add a tag for “sentence ending”?

I'm new to Corpus Linguistics and I'm writing a paper about the English and Portuguese "because noun", a type of construction such as "I'm going home because GTA5". However, when I try to search this ...
0
votes
0answers
61 views

Vocabulary Wordlist for the elementary school English

I am curious if there is a written English corpus that offers word frequency lists for the elementary stage vocabulary, i.e. K-5 grades? Something like Dr. Edward Fry's sight words, which is updated ...
3
votes
1answer
420 views

Is the Philippine corpus known as Palito still available for download online?

I have visited http://ccs.dlsu.edu.ph:8086/Palito/, but it is no longer online. SEAlang has the same content, I think, but it isn't downloadable. Is it still available for download online?
9
votes
3answers
1k views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...