Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

2
votes
1answer
47 views

Does anyone have a link to the American Local News Corpus V1.0?

I found a paper http://www.cs.jhu.edu/~anni/papers/alnc_lrec14.pdf that has made this epic text corpus of over 1 billion words available somewhere, but I can't find it anywhere online. The paper says ...
0
votes
1answer
23 views

Need an online freely available Anaphorically Annotated Corpus of English Language for Identification of Discourse Units

I'm in need of a freely available corpus of English language (which may contain text of any genre) but has to be annotated anaphoriacally i-e. Anaphora resolution should have been performed already. ...
3
votes
0answers
35 views

Is there an english news corpus available to download for between 1900 and 201X (free or low cost)

I'm attempting a word embedding analysis (think underlying meaning and implications, but computational) of certain keywords through time in the English language, but I am having some difficulty ...
4
votes
2answers
241 views

How to interpret this form of Heaps' Law?

Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia ...
1
vote
1answer
26 views

Is there a way to search for language data by context?

For example, if I wanted to see real-life transcriptions of dialogue in situations like a date, a business meeting, two friends walking in a park, etc, just to be able to generate lists of particular ...
3
votes
1answer
39 views

Are there any corpora of informal and unstructured text labelled for Named Entity Recognition?

I have been searching since last week for annotated informal texts (with a lot of misspelled words, slang, etc.) to test some Named Entity Recognition tools for research purposes. For example, it ...
0
votes
1answer
24 views

Are there corpora which tag phrases by discourse- or conversational-function?

More generally, imagine a corpus which could identify the conversational function of a sentence, such as, "Have a nice day!", in narrow categories like, "ending a conversation", or broad ones like "...
2
votes
3answers
76 views

Extensive English word corpora

I am looking for an extensive list of english words[including American and British... just an exhaustive list]. This list of english words should contain an exhaustive collection of all forms of all ...
2
votes
1answer
39 views

register variation: unbalanced corpus sample

I'm doing a small pilot study on evidentiality in a language I work on. I'm looking at two evidential markers, -mi that is said to mark the source of knowledge as personal, and -shi that marks ...
2
votes
0answers
57 views

Collocation: interpreting contingency table for log-likelihood measure

I have a conundrum here. I'm attempting to recreate the log-likelihood test example in 'The Cambridge Handbook of Corpus Linguistics (2015). However when I run the same test on the same data I get a ...
4
votes
0answers
41 views

PoS tagger - technical corpus

I try to find corpus with technical language to train my model, but I cannot find it. Do you know if there are any tagged PoS corpus in English related with technical language?
6
votes
1answer
90 views

How can I know if a log-likelihood score is high enough?

I am trying to understand if some terms can be considered as distinctly American or British. So for example I was studying the term "aubergine" and the normalized frequency in the American corpus is ...
5
votes
1answer
65 views

Word length as a function of word frequency?

I was wondering if any studies have been done in corpus linguistics which look at word length (number of phonemes per word) as a function of the word's frequency of use? Thanks! EDIT: Never mind, ...
2
votes
0answers
18 views

How to convert CHAT to TextGrid in a batch?

Siarad corpus is a corpus of spoken Welsh (~40 hours, all transcribed, glossed and translated!). I want to analyze some conversations using Praat. Praat uses the TextGrid format, but the Siarad corpus ...
4
votes
1answer
31 views

Looking for Spanish email or chat corpora

I am looking for some corpora containing emails or chats in Spanish.
1
vote
3answers
78 views

Words with the hightest frequency that are not Collocates of each other (inside a text corpora)

Is there a way to obtain a: List, that has 2 words per row Each 2 words are not collocates (or collocations) of each other (ie: are not found within the same phrase inside the entire text corpus, or ...
1
vote
1answer
62 views

Dataset of subject-verb-object/adverb-verb/adjective-noun pairs

Does there exist a dataset which contains a list of subject-verb-object triplets or adverb-verb/adjective-noun pairs that occur within a certain (large) corpus? If not, what would be the most ...
2
votes
1answer
47 views

How to extract example sentences from linguistics papers

Does anyone know of a good tool to grab all example sentences used in a paper so they could be processed via an NLP pipeline? I have udpipe in mind especially: I'd like to take sentences from one or ...
0
votes
0answers
28 views

Corpora with graph data base like neo4j

Relating with corpus linguistics with graph databases, I've seen some examples of using neo4j for store the words of a single phrase in which it is used a relation of --NEXT--> to link one word with ...
4
votes
1answer
66 views

JSON for corpora

Do you know any cases of corpus linguistics tools or corpora, using JSON to "mark" or delimited linguistics units in morphology, syntax,semantics? It seems all marking systems use some kind of XML or ...
0
votes
0answers
24 views

regression analysis

A fellow student advised me, that to make predictions based on diachronic corpus data, one should use the predict formula in Rstudio and create a regression analysis. Can someone please confirm this? ...
1
vote
2answers
103 views

Corpus for simple or basic english

I'm trying to find a good, extensive corpus that uses "simple" or "basic" English, in the sense that it should be easily understandable to people who are not proficient in the English language. For ...
3
votes
2answers
86 views

Which free resource to find word frequency?

My hypothesis is that math jargon is more aliening than other academic fields, and I want to concrete that by an example of the average frequency of the word in these two definitions of "...
2
votes
1answer
39 views

When corpus data suggests a word is common in a language?

I have two questions. If I were to use diachronic corpora to investigate the frequency of usage  for  words within  a word class of English that is mostly archaic today, how do I determine if the ...
1
vote
1answer
40 views

What statistical test is appropriate in a corpus analysis of pronoun resolution preferences?

I have extracted data from a corpus that collected sentences with an antecedent and a next sentence with a pronoun or a stressed pronoun in subject position. Now, the anaphora can either refer back to ...
5
votes
0answers
63 views

Is there a threshold of usage of a word, below which it will become an archaism?

University student, beginning to learn about corpora, developing a research question for an assignment. There may well be linguistic concepts and terminology that already cover this idea and that I am ...
2
votes
1answer
135 views

What is the state of the art for diacritic restoration?

Diacritic restoration is a term for guessing the original form of a text, after it's been converted to ASCII in a lossy way. In Lingála, for example, writing in ASCII merges two pairs of vowels and ...
1
vote
0answers
57 views

Mandarin Chinese and Antconc

Can Antconc be used to build a corpus for Mandarin, Chinese? Specifically, could characters be recognized, or if one were to upload a pinyin text, could tones be displayed? Is this possible?
0
votes
0answers
23 views

How to determine word frequency based on demographic? (Does a public library of word frequencies exist by demographic)

I'm not sure "sign post" questions are appropriate however that is what this is. I am looking for a tool, a means of answering questions such as "Word X is used more by demographic Y than demographic ...
3
votes
0answers
53 views

Is there a frequency-ranked corpus of Punjabi NAV lemmata?

I am looking for a list of lexical roots for Punjabi, with frequency information whereby one can get an idea what the 1000 most-frequent words of the language are. I need to exclude case forms etc. ...
2
votes
2answers
74 views

glossary/dictionary corpus for NLP task

I am looking for glossary/dictionary corpus in machine readable form. Basically huge file/db which would contain word and its definition. Here is sample: Authenticity: a feature that characterizes ...
3
votes
1answer
153 views

Phonetically annotated speech corpus

Are there any phonetically annotated corpora of accented English speech? Preferably English spoken by native English speakers with a strong accents, such as speakers from a specific region in the UK ...
2
votes
1answer
56 views

where can I find free corpus of spoken disabled people (in english, italian)

I am working on the research project in NLP domain. Basically, we are looking for a free audio corpus of spoken disabled people to study (for instance, aged people, or people has difficulty in normal ...
4
votes
2answers
91 views

Tools and algorithms for finding non-verbatim quotations

I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite ...
1
vote
0answers
25 views

Online data giving numbers and frequencies of nouns in some well-known novels?

Are there any freely available online data giving the numbers and frequencies of nouns used in some well-known novels, whether or not the novels are in the public domain? I mean data such as Dan ...
0
votes
0answers
39 views

Corpus Assisted Discourse Studies

I wanted a corpus of recent English fiction (20 century onwards) which I can use as a reference corpus for my research work. It should consist of at least 5 million words. I do not need it in ...
1
vote
0answers
34 views

Corpora for sentence compression

I am looking for corpora for sentence compression. I am mostly interested in English, and any type of sentence. I am aware of: https://github.com/google-research-datasets/sentence-compression ...
1
vote
1answer
31 views

Corpora for extractive summaries

I am looking for corpora of extractive summaries. I am mostly interested in English, and any type of document.
0
votes
1answer
2k views

What is the difference between type and token?

My understanding of this is quite vague. Token I understand to be the total number of words in a given text, but type I am not so sure about. If I have a variety of inflected forms (e.g eat, eats, ...
2
votes
2answers
62 views

Command line tools for querying corpuses

I've built myself a little corpus that I'm using for language-learning. However, I'm reaching the limits of standard command-line tool-chain: grep and friends. For example, I might look for ...
1
vote
1answer
37 views

Software for exploring large set of textual data

I am working with large amounts of text and already have a bunch of programmatic instruments to explore them (Python, R, etc). To say the truth, most of the exploration tasks like bigram extraction, ...
2
votes
0answers
196 views

Parallel English-Spanish corpora?

I would like to know of parallel corpora for English-Spanish (that is, texts or language originally in English translated into Spanish and aligned by excerpts of texts) other than those on the Opus ...
4
votes
3answers
820 views

Where can I find a free text corpus for the Hindi language?

I need a text corpus for my NLP project in Hindi, however all the corpora I found are not free for downloading. Is there any free annotated Hindi text corpus at all? If yes, where can I find it?
0
votes
0answers
44 views

Vocabulary Wordlist for the elementary school English

I am curious if there is a written English corpus that offers word frequency lists for the elementary stage vocabulary, i.e. K-5 grades? Something like Dr. Edward Fry's sight words, which is updated ...
1
vote
0answers
42 views

Is there any free, single-word wordlist with CLAWS7 part-of-speech tags?

I am looking for a wordlist which includes single-word entries and their part-of-speech tags in UCREL CLAWS7 format. The list must be comprised of scientifically collected data, for example words ...
1
vote
0answers
15 views

corpus search in Sketch Engine for particular correlative coordinators

Can someone teach me/show me how to search for either...or, neither...or/nor, whether...or and etc. in Sketch engine (in BAWE corpus particularly)? I have read basically every guide and watched a lot ...
1
vote
0answers
31 views

Error of word frequency [closed]

I have created a database of lyrics with some 10000 songs and did a word frequency analysis to find the most common words. Next, i sorted the corpus according to the publication date and measured the ...
1
vote
1answer
48 views

Are there any computer readable corpora for ASL?

I'm looking to see if there are any free-to-use corpora in ASL (any domain is fine), transcribed so a computer can use it. I am interested in doing some analysis on ASL syntax, morphology etc. But I ...
3
votes
1answer
93 views

Classifying the verbs in a small corpus

I have a relatively small language corpus (ca. 250000 tokens). There are two possible stems for a verb in the language: active and passive. Some of the verbs in the language occur only in passive ...
4
votes
1answer
158 views

Where can I find training data for dialects of Hindi?

I am working on a NLP project that aims at identifying different dialects of Hindi language. My über goal is to generate dialogs in once the program identifies which dialect is given. I tried ...