Questions tagged [corpora]
A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.
242
questions
0
votes
0
answers
24
views
Random sample of the search result?
I am looking at how the word maybe is used in COCA. As I
want to analyze the contexts it occurs in, I need to random-sample the search result (which is around 374K hits), so that I can see in what ...
- 11
1
vote
0
answers
34
views
Is there a way for random term extraction when searching COCA (the Corpus of Contemporary American English)?
When I search for 1000 examples of maybe using COCA data, most of the data that comes up is from older years.
How can I obtain data that is randomly extracted based on the era in which it was used, ...
- 11
0
votes
1
answer
43
views
Ideal number of randomized concordancing lines
I want to perform randomized concordancing on a news corpus with 807 mil. tokens from 2 mil. documents from 2011-2022. Is there any ideal number of concordancing in order to be able to make a ...
- 43
0
votes
0
answers
17
views
Quantative/Statistical comparison between unequal corpora
I have created a corpus of 400.000 words, consisting exclusively of governmental administrative documents.
I am focusing on the usage of rare words and i want to prove that my corpus has increased ...
0
votes
0
answers
24
views
Is plain orthographic transciption of speech recordings adequate for corpus creation?
NOTE: I am not a linguist, but my interest is in the Natural Language Processing (NLP) field.
I am doing a research which is focused only on gathering new or distinct words on a recently described ...
- 101
0
votes
0
answers
22
views
Are open-ended questions allowed for an interview focused on corpus creation?
NOTE: I am not a linguist, but my interest is in the Natural Language Processing (NLP) field.
I am doing a research which is focused only on gathering new or distinct words on a recently described ...
- 101
1
vote
0
answers
40
views
"data cleaning" of tatoeba sentence file
I'm trying to automate Anki deck creation for my language learning. To this end, I downloaded a file with sentences from Tatoeba. My goal is to calculate the frequency ranking of each word in every ...
- 29
-1
votes
2
answers
79
views
Do I need to lemmatize my corpus when performing collocation analysis on inflective languages?
Do I need to lemmatize my corpus when I am going to perform a collocational analysis? Several language, including English, are to a certain extent inflective. These inflections sometimes carries ...
- 43
0
votes
0
answers
63
views
Using semantic preference as method for critical linguistics research?
I am intending to conduct a critical linguistics research using corpus linguistics as a method, particularly, the analysis of semantic preference analysis to word-pair patterns (henceforth collocates)....
- 43
1
vote
0
answers
70
views
What differentiates the usage of MI and MI3 in collocation analysis?
In Brezina (2018), it is stated that to measure collocation strength, one may use the Mutual Information (MI) to test the collocational strength of a given collocation. MI is naturally more sensitive ...
- 43
0
votes
1
answer
36
views
What does the word "company" mean in the context of collocation analysis in corpus linguistics?
I am currently reading Brezina (2018) on Statistics in Corpus Linguistics. I have a hard time trying to decipher what the author means of "company" in this sentence:
Dice and log Dice, and ...
- 43
1
vote
1
answer
34
views
Why use log per-million count in this analysis of Brevity Law using corpora?
I am curious about corpus linguistics, and especially in this case how corpora is used to prove the Brevity Law, also known as Zip'f law of abbreviation. Simply put, this theorem postulates that the ...
- 43
3
votes
0
answers
66
views
Corpus studies on the frequency of subject questions in English
Are there any corpus studies which show the relative frequency of different types of interrogative main clauses in English, in particular the relative frequency of subject questions (which do not ...
- 31
2
votes
0
answers
33
views
Dealing with words having token-frequency 1 when calculating Phonotactic Probability
NOTE
This is a repost of a question on the Mathematics StackExchange Board. It has been modified to fit this board :)
I am open to suggestions
Explanation
I am trying to calculate phonotactic ...
- 121
0
votes
1
answer
55
views
Corpus Linguistics Major [closed]
I want to use corpus in my MA research, but I have a large problem; how can I analyze the data? Do I use software corpora or corpus such as BNC? I know it is according to my research question, but ...
- 19
-2
votes
1
answer
68
views
How could you study “cultural differences” using corpora? [closed]
Anyone who’s traveled has come upon a sense of there being some “cultural differences”, but my mind is always seeking to systematize things, and I’ve wondered if there is a better way to collect tons ...
- 590
10
votes
2
answers
479
views
Are linguistic corpora under threat from AI-generated text?
As many will be aware, there has been a lot of concern about the spread of AI generated text posted on the Stack Exchange network, leading to moderators on several sites standing down. See here and ...
- 4,058
1
vote
2
answers
191
views
Should I use normalized frequencies or raw frequencies when conducting Log-likelihood test to compare two corpora with differing text lenghts?
Analyzing multiword expressions and I want to compare two corpora of similar sizes, but these corpora differ in average text length. For example average text length in Corpus A. is 300 while it is 600 ...
2
votes
0
answers
88
views
COCA corpus: Is it possible to display a list search result as parts of speech?
I want to know what parts of speech (and in what frequency) follow a word or phrase. If I do a list search, for example "for the purpose of *", the result is a list of the actual words in ...
1
vote
1
answer
82
views
Where to find English POS words?
Given a POS (part of speech), tag, and dependency for words, where can I find words that have the same POS, tag, and dependency?
I only need a list of words by (pos,tag) and do not need code for this (...
- 21
0
votes
1
answer
58
views
I need help with searching a corpus for a complex sentence
I am trying to formulate a search request that will show me example sentences for sentences that contain the phrase "In the beginning" or "At the beginning" followed by the word &...
- 11
0
votes
0
answers
38
views
Is there any mathematically studied ontology space?
I am considering “ontology generation”. I have not yet read the specifics of these techniques.
Still, the point must surely be to identify some kind of cooccurrences / associations between words. ...
- 590
0
votes
0
answers
41
views
Where can I find a free/cheap corpus of the 20,000 most frequently used words in the English language? [duplicate]
There are a bunch of large corpora containing millions or billions of words.
My needs are simpler: I only need the 20K most used words. Preferably, these words should be most used in the general ...
0
votes
1
answer
34
views
Comparing Text Translations by Conducting Corpus Analysis
Provided one has three pieces of text called - “Text A”, “Text B” and “Text C”.
Text A is the original text.
Text B and Text C are translations of Text A (Text A, being initially written in another ...
1
vote
1
answer
70
views
Can someone share with me an article that studies the decline of by-phrases in the passive?
Can someone share with me an article that studies the decline of by-phrases in the passive? Preferably a corpus-based study, but this is not that relevant
user36925
3
votes
0
answers
46
views
Quantitative methodology for contrastive pragmatics in corpus-based settings
I am interested in literature regarding methodology that could be relevant for quantitative research into differences in pragmatic meaning between two 'equivalent' concepts in two languages (in other ...
1
vote
0
answers
60
views
What is a good multi language translation system for individual parts of speech such as verbs, nouns, adjectives?
I am developing a language learning tool in Python that generates dual-language books intended to be read as audio books.
The system should work by giving single word translations after every ...
- 11
0
votes
1
answer
76
views
Are alveolo-palatal consonants more likely to be followed by high vowels, whereas retroflex consonants are more likely followed by low vowels?
It seems to me that high voles like i would more naturally follow alveolo-palatal consonants because the need to "spread the lips" (in the popular description of the latter) seem to more ...
5
votes
3
answers
1k
views
How can I find the frequency for the Russian word "чем-нибудь"?
I've tried searching for the word "чем-нибудь"on the 20,000 word list I found on Wikipedia. You can link directly to it here. For the sake of thoroughness, I even searched the larger 50,...
- 313
0
votes
1
answer
334
views
Can someone explain to me the Zipf–Mandelbrot law?
I really can't understand. It's about linguistics and I can't understand anything because there are mathematical formulas in it that I can't understand at all. Can anyone explain this with ...
- 35
3
votes
1
answer
62
views
Bilingual corpora work from the late 1980s
Does anyone know if the paper by Warwick and Russell (often cited as below) exists? I have looked on the EURALEX website and the authors are not listed as speakers in the 4th Conference. Maybe the ...
- 171
2
votes
1
answer
134
views
Has anyone ever ranked the prevalence of phones by number of speakers worldwide?
I'm interested in knowing the most-used and least-used phones worldwide. According to Wikipedia, the IPA charts about 140 pulmonic consonants, 80 non-pulmonic consonants, 30 co-articulated consonants, ...
- 121
0
votes
1
answer
107
views
What is the frequency of color words in English compared to German?
I am writing a term paper about the role of adjectives and gender marking in efficient communication about English and German respectively.
And I was wondering if there is any way/website/tool to ...
1
vote
1
answer
51
views
Large parallel corpus of English and Farsi text
hope you are doing well.
I am looking for a large parallel corpus of English and Farsi text, professionally written or edited. I would be more than happy to have your suggestions.
Thank you all.
- 13
0
votes
1
answer
45
views
Where can I find a corpus of sentence with the resulting emotion in french?
I'm working on a sentiment analysis program and I need to validate it using a corpus of sentence with the linked emotions.
I need to perform this task on a french corpus and I can't find any on the ...
- 101
2
votes
1
answer
84
views
Are there any set guidelines for the lifespan of a (spoken) corpus?
Are there any set guidelines on how long a (spoken) corpus stays relevant / is reliable to use? The only major Dutch spoken corpus is from 2004, and I was wondering if using this corpus would cause a ...
0
votes
1
answer
376
views
Where can I get the complete list of word entries present in Merriam Webster's dictionary?
I can't find any list, github repo, etc to find the total list of entries defined in merriam webster. Any ideas?
- 109
11
votes
2
answers
839
views
How to best clean a large historical corpus ridden with OCR errors
Overview:
I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some ...
- 231
1
vote
0
answers
42
views
Corpus of Taboo card game?
Has anyone constructed a corpus of people playing the card game Taboo, in a purely text-based setting, like a computer chat, rather than face-to-face with audio or video? The closest I've found are ...
- 576
3
votes
1
answer
153
views
How order of the syntax tree is formally/strictly proven?
An example from https://en.wikipedia.org/wiki/Concrete_syntax_tree : tree of "John hit the ball." .
The tree branches are joined in this order:
{John {hit {the ball}}}.
Ie, "the" ...
- 131
5
votes
1
answer
1k
views
Where online can I find a list of all the Hapax Legomena in the Hebrew Bible?
I am looking for a list online of all Hapax Legomena in the Hebrew Bible in text format that I can copy and paste into MS Word for further studies.
I need something that's free and is not in a picture ...
1
vote
2
answers
87
views
Is it possible to test statistical significance of difference between two distance/similarity scores?
I have a corpus consisting of tweets by men and another corpus of female tweets. I was thinking of using a word embeddings approach (e.g., glove and fasttext) and examine the cosine similarity between ...
- 121
2
votes
0
answers
63
views
Is it possible to infer the meaning of a word of a language based on corpus analysis, without prior knowledge of the language?
If I am totally foreign to a language, are there corpus analysis methodologies and theories that I can employ to figure out the meaning of a word in a corpus based on that language?
If yes, do point ...
- 347
1
vote
1
answer
142
views
Praat - window size larger than vowel
I'm doing an acoustic-phonetic analysis of read-speech in an American English corpus.
If we want to set a window/frame size (25 - 30 ms) to cover the majority of vowels in the corpus, but some vowels (...
- 11
2
votes
0
answers
86
views
Word alignment for Switchboard Dialogue Acts Corpus
I am trying to connect the dialogue acts of the Switchboard Dialogue Acts Corpus with the word alignment timing information available here.
I can see how it is possible to match up every utterance ...
- 576
2
votes
0
answers
228
views
Is the source list of "Oxford English Corpus" public available?
The Oxford English Corpus (https://en.wikipedia.org/wiki/Oxford_English_Corpus) is a text corpus of 21st century English, used by the makers of the Oxford English Dictionary and by Oxford University ...
- 121
1
vote
2
answers
248
views
How can I search a corpus for easy sentences?
I would like to search a large corpus for example sentences which contain exclusively or a high proportion of words from the N most common words in some frequency dictionary (the language of the ...
- 179
1
vote
0
answers
127
views
What word category do twink, adonis, muse etc. belong to? [closed]
Terms such as candy, cutie, honey, princess, diamond, queen(?), stud and bunny are terms of endearment (these terms are often used in a relationship to show affection and may also be used, with ...
- 111
1
vote
1
answer
131
views
Spoken Arabic dictionary/corpus?
Is there an existing corpus or dictionary for different spoken Arabic dialects? I'd like to look up a word (in English or MSA or in a spoken dialect) and find where this word or phrase is spoken (and ...
- 191
6
votes
1
answer
89
views
Corpus of English sentences written by non-native speakers?
I am working on a hobby project which tends to help the non-native speakers figure out if the sentences they write are good enough and sounds like written by English native speakers. Any relevant info ...
- 163