Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
5
votes
1answer
33 views

Corpus of English sentences written by non-native speakers?

I am working on a hobby project which tends to help the non-native speakers figure out if the sentences they write are good enough and sounds like written by English native speakers. Any relevant info ...
-1
votes
1answer
17 views

Which is the most recommended corpus for training a POS Tagger?

The tagger is for English and I will use the universal dependency tag set. I am looking for a corpus that best represents nowadays English.
0
votes
0answers
23 views

What English corpus for syntactic parsing should I work with?

I'm working on a syntactic parser (constituent structure based) and I need a corpus with just sentences, not phrases. What corpus should I use? Is there a corpus tagged with the Universal Dependency ...
0
votes
1answer
32 views

Comparing corpora

So first, a disclaimer: I'm less than beginner level in everything that has to do with programming. I'm analysing the occurrences of English code-switching/translanguaging in a particular Polish-...
0
votes
0answers
10 views

English corpus containaing simple and complex sentences [closed]

I want a corpus that contains a list of sentences by newline(preferably) and that contains simple and complex sentences.
2
votes
0answers
31 views

Searching for an English Whats-app Corpus

I am searching an English Whats-app Corpus in order to analyse a linguistic phenomena. I had some difficulties to find one and maybe some of you can help me out. It is only for corpus driven study and ...
1
vote
0answers
37 views

Where are some of the biggest freely available English 1 on 1 conversation corpuses that are in plain text preferably?

I'm looking for a free, preferably plain text, 1 on 1 English casual conversation (such as texting back and forth) corpus that requires little to no preprocessing (finding and replacing characters). ...
0
votes
0answers
33 views

L2 Polish writing database

I'm looking for a database of L2 Polish writing containing written output of learners of Polish as a foreign language. Perhaps anyone knows where I may be able to find this kind of info?
0
votes
0answers
36 views

How to determine a framework? [closed]

I will soon be writing my thesis (a corpus analysis on dating websites) in which I will investigate, through quantative analysis, how textual-self presentation between men and women is achieved. I ...
6
votes
1answer
67 views

How to make a Corpus? (selecting a good structure)

I'm making a Corpus. I have searched for good practices on how to build it. I've downloaded some Corpus to check their internal structure and the extension file, but I didn't find many. So, I am lost ...
1
vote
1answer
44 views

Suggestion for software to analyze data from social media- please advise

I am searching urgent for a software that I can use to analyse corpora. In short, I would like to have a look at collocations, ellipsis, frequencies, concordances etc. I also want to search for themes ...
5
votes
1answer
99 views

What are the conventions for corpus usage?

Suppose I'm working with a language that doesn't have much data available; the largest corpus around is a translation of the Bible, which is free to read online, but is presumably still copyrighted by ...
1
vote
1answer
44 views

Is there a comprehensive list of corpora of all the languages (maybe even dialects) of the world?

Is there a comprehensive list of corpora of all the languages of the world? I've been searching on wikipedia eg. https://en.wikipedia.org/wiki/Czech_National_Corpus. If I scroll to the bottom, I do ...
2
votes
1answer
65 views

Does anyone have a link to the American Local News Corpus V1.0?

I found a paper http://www.cs.jhu.edu/~anni/papers/alnc_lrec14.pdf that has made this epic text corpus of over 1 billion words available somewhere, but I can't find it anywhere online. The paper says ...
0
votes
1answer
27 views

Need an online freely available Anaphorically Annotated Corpus of English Language for Identification of Discourse Units

I'm in need of a freely available corpus of English language (which may contain text of any genre) but has to be annotated anaphoriacally i-e. Anaphora resolution should have been performed already. ...
3
votes
0answers
39 views

Is there an english news corpus available to download for between 1900 and 201X (free or low cost)

I'm attempting a word embedding analysis (think underlying meaning and implications, but computational) of certain keywords through time in the English language, but I am having some difficulty ...
4
votes
2answers
252 views

How to interpret this form of Heaps' Law?

Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia ...
1
vote
1answer
26 views

Is there a way to search for language data by context?

For example, if I wanted to see real-life transcriptions of dialogue in situations like a date, a business meeting, two friends walking in a park, etc, just to be able to generate lists of particular ...
3
votes
1answer
43 views

Are there any corpora of informal and unstructured text labelled for Named Entity Recognition?

I have been searching since last week for annotated informal texts (with a lot of misspelled words, slang, etc.) to test some Named Entity Recognition tools for research purposes. For example, it ...
0
votes
1answer
27 views

Are there corpora which tag phrases by discourse- or conversational-function?

More generally, imagine a corpus which could identify the conversational function of a sentence, such as, "Have a nice day!", in narrow categories like, "ending a conversation", or broad ones like "...
2
votes
3answers
83 views

Extensive English word corpora

I am looking for an extensive list of english words[including American and British... just an exhaustive list]. This list of english words should contain an exhaustive collection of all forms of all ...
2
votes
1answer
52 views

register variation: unbalanced corpus sample

I'm doing a small pilot study on evidentiality in a language I work on. I'm looking at two evidential markers, -mi that is said to mark the source of knowledge as personal, and -shi that marks ...
2
votes
0answers
100 views

Collocation: interpreting contingency table for log-likelihood measure

I have a conundrum here. I'm attempting to recreate the log-likelihood test example in 'The Cambridge Handbook of Corpus Linguistics (2015). However when I run the same test on the same data I get a ...
4
votes
0answers
46 views

PoS tagger - technical corpus

I try to find corpus with technical language to train my model, but I cannot find it. Do you know if there are any tagged PoS corpus in English related with technical language?
6
votes
1answer
154 views

How can I know if a log-likelihood score is high enough?

I am trying to understand if some terms can be considered as distinctly American or British. So for example I was studying the term "aubergine" and the normalized frequency in the American corpus is ...
5
votes
1answer
69 views

Word length as a function of word frequency?

I was wondering if any studies have been done in corpus linguistics which look at word length (number of phonemes per word) as a function of the word's frequency of use? Thanks! EDIT: Never mind, ...
2
votes
0answers
26 views

How to convert CHAT to TextGrid in a batch?

Siarad corpus is a corpus of spoken Welsh (~40 hours, all transcribed, glossed and translated!). I want to analyze some conversations using Praat. Praat uses the TextGrid format, but the Siarad corpus ...
4
votes
1answer
33 views

Looking for Spanish email or chat corpora

I am looking for some corpora containing emails or chats in Spanish.
1
vote
3answers
86 views

Words with the hightest frequency that are not Collocates of each other (inside a text corpora)

Is there a way to obtain a: List, that has 2 words per row Each 2 words are not collocates (or collocations) of each other (ie: are not found within the same phrase inside the entire text corpus, or ...
1
vote
1answer
78 views

Dataset of subject-verb-object/adverb-verb/adjective-noun pairs

Does there exist a dataset which contains a list of subject-verb-object triplets or adverb-verb/adjective-noun pairs that occur within a certain (large) corpus? If not, what would be the most ...
2
votes
1answer
51 views

How to extract example sentences from linguistics papers

Does anyone know of a good tool to grab all example sentences used in a paper so they could be processed via an NLP pipeline? I have udpipe in mind especially: I'd like to take sentences from one or ...
4
votes
2answers
141 views

JSON for corpora

Do you know any cases of corpus linguistics tools or corpora, using JSON to "mark" or delimited linguistics units in morphology, syntax,semantics? It seems all marking systems use some kind of XML or ...
1
vote
2answers
159 views

Corpus for simple or basic english

I'm trying to find a good, extensive corpus that uses "simple" or "basic" English, in the sense that it should be easily understandable to people who are not proficient in the English language. For ...
3
votes
2answers
99 views

Which free resource to find word frequency?

My hypothesis is that math jargon is more aliening than other academic fields, and I want to concrete that by an example of the average frequency of the word in these two definitions of "...
2
votes
1answer
41 views

When corpus data suggests a word is common in a language?

I have two questions. If I were to use diachronic corpora to investigate the frequency of usage  for  words within  a word class of English that is mostly archaic today, how do I determine if the ...
1
vote
1answer
41 views

What statistical test is appropriate in a corpus analysis of pronoun resolution preferences?

I have extracted data from a corpus that collected sentences with an antecedent and a next sentence with a pronoun or a stressed pronoun in subject position. Now, the anaphora can either refer back to ...
6
votes
0answers
66 views

Is there a threshold of usage of a word, below which it will become an archaism?

University student, beginning to learn about corpora, developing a research question for an assignment. There may well be linguistic concepts and terminology that already cover this idea and that I am ...
2
votes
1answer
144 views

What is the state of the art for diacritic restoration?

Diacritic restoration is a term for guessing the original form of a text, after it's been converted to ASCII in a lossy way. In Lingála, for example, writing in ASCII merges two pairs of vowels and ...
1
vote
0answers
75 views

Mandarin Chinese and Antconc

Can Antconc be used to build a corpus for Mandarin, Chinese? Specifically, could characters be recognized, or if one were to upload a pinyin text, could tones be displayed? Is this possible?
0
votes
0answers
26 views

How to determine word frequency based on demographic? (Does a public library of word frequencies exist by demographic)

I'm not sure "sign post" questions are appropriate however that is what this is. I am looking for a tool, a means of answering questions such as "Word X is used more by demographic Y than demographic ...
3
votes
0answers
58 views

Is there a frequency-ranked corpus of Punjabi NAV lemmata?

I am looking for a list of lexical roots for Punjabi, with frequency information whereby one can get an idea what the 1000 most-frequent words of the language are. I need to exclude case forms etc. ...
2
votes
2answers
101 views

glossary/dictionary corpus for NLP task

I am looking for glossary/dictionary corpus in machine readable form. Basically huge file/db which would contain word and its definition. Here is sample: Authenticity: a feature that characterizes ...
3
votes
1answer
231 views

Phonetically annotated speech corpus

Are there any phonetically annotated corpora of accented English speech? Preferably English spoken by native English speakers with a strong accents, such as speakers from a specific region in the UK ...
2
votes
1answer
59 views

where can I find free corpus of spoken disabled people (in english, italian)

I am working on the research project in NLP domain. Basically, we are looking for a free audio corpus of spoken disabled people to study (for instance, aged people, or people has difficulty in normal ...
4
votes
2answers
91 views

Tools and algorithms for finding non-verbatim quotations

I have a textual corpus consisting of about 500 different texts, written in the time span of several centuries. The authors of these texts belonged to the same culture, so they used to read and cite ...
1
vote
0answers
25 views

Online data giving numbers and frequencies of nouns in some well-known novels?

Are there any freely available online data giving the numbers and frequencies of nouns used in some well-known novels, whether or not the novels are in the public domain? I mean data such as Dan ...
0
votes
0answers
39 views

Corpus Assisted Discourse Studies

I wanted a corpus of recent English fiction (20 century onwards) which I can use as a reference corpus for my research work. It should consist of at least 5 million words. I do not need it in ...
1
vote
0answers
39 views

Corpora for sentence compression

I am looking for corpora for sentence compression. I am mostly interested in English, and any type of sentence. I am aware of: https://github.com/google-research-datasets/sentence-compression ...
1
vote
1answer
31 views

Corpora for extractive summaries

I am looking for corpora of extractive summaries. I am mostly interested in English, and any type of document.
0
votes
1answer
3k views

What is the difference between type and token?

My understanding of this is quite vague. Token I understand to be the total number of words in a given text, but type I am not so sure about. If I have a variety of inflected forms (e.g eat, eats, ...