Questions tagged [corpora]
A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.
209
questions
1
vote
1answer
25 views
Are there any set guidelines for the lifespan of a (spoken) corpus?
Are there any set guidelines on how long a (spoken) corpus stays relevant / is reliable to use? The only major Dutch spoken corpus is from 2004, and I was wondering if using this corpus would cause a ...
2
votes
2answers
104 views
register variation: unbalanced corpus sample
I'm doing a small pilot study on evidentiality in a language I work on.
I'm looking at two evidential markers, -mi that is said to mark the source of knowledge as personal, and -shi that marks ...
1
vote
1answer
69 views
Is there any free, single-word wordlist with CLAWS7 part-of-speech tags?
I am looking for a wordlist which includes single-word entries and their part-of-speech tags in UCREL CLAWS7 format.
The list must be comprised of scientifically collected data, for example words ...
3
votes
7answers
7k views
Sounds that humans make with their mouths
I'm interested in a list of sounds that humans make with their mouths.
I'm not looking for
speech sounds
sounds that take a non-average skill (such as beatboxing)
sounds that cannot be made with the ...
8
votes
7answers
3k views
Biggest freely available English corpus?
Any help on finding the biggest freely available English corpus that can be used on research?
So far I have found OANC with 15 M words.
0
votes
1answer
35 views
Where can I get the complete list of word entries present in Merriam Webster's dictionary?
I can't find any list, github repo, etc to find the total list of entries defined in merriam webster. Any ideas?
11
votes
2answers
589 views
How to best clean a large historical corpus ridden with OCR errors
Overview:
I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some ...
0
votes
0answers
36 views
Potsdam Sentence Corpus - English comparable corpora
Dear fellow Linguists,
I am searching for an English corpus that has a high degree of comparability with the German Potsdam Sentence Corpus. Naturally I am already reading a lot of literature, but I ...
1
vote
0answers
33 views
Corpus of Taboo card game?
Has anyone constructed a corpus of people playing the card game Taboo, in a purely text-based setting, like a computer chat, rather than face-to-face with audio or video? The closest I've found are ...
3
votes
1answer
82 views
How order of the syntax tree is formally/strictly proven?
An example from https://en.wikipedia.org/wiki/Concrete_syntax_tree : tree of "John hit the ball." .
The tree branches are joined in this order:
{John {hit {the ball}}}.
Ie, "the" ...
4
votes
2answers
313 views
How to interpret this form of Heaps' Law?
Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia ...
2
votes
2answers
367 views
Where to find/purchase an offline (downloadable) Arabic corpus
I'm developing an application to teach Quran. I need some textual/lingual analysis based on a common and general corpus.
I know about King Abdulaziz City for Science and Technology (KACST) Arabic ...
5
votes
3answers
601 views
Are there tools to build domain specific corpus? [closed]
I would like to build a domain specific corpus. By this I mean a corpus from a specific knowledge area, for instance Law, Psychiatry, or Klingon language.
I am currently using a Wikipedia corpus, and ...
7
votes
2answers
928 views
Is there any corpus for technical English? (E.g., computers, IT, modern technology)
Alternatively, do any larger corpora have categories that pertain to technology, computer science, etc.? Or collections of English documentation and/or manuals related to modern devices, interfaces, ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
0
votes
0answers
16 views
Determining corpora size and participant count
I am setting up a linguistics experiment, involving subjective judgements of synthetic speech. For the sake of example, let's say I'm testing 5 systems, using a Likert scale of 1-5 as a rating. How ...
4
votes
1answer
185 views
Where online can I find a list of all the Hapax Legomena in the Hebrew Bible?
I am looking for a list online of all Hapax Legomena in the Hebrew Bible in text format that I can copy and paste into MS Word for further studies.
I need something that's free and is not in a picture ...
1
vote
2answers
59 views
Is it possible to test statistical significance of difference between two distance/similarity scores?
I have a corpus consisting of tweets by men and another corpus of female tweets. I was thinking of using a word embeddings approach (e.g., glove and fasttext) and examine the cosine similarity between ...
0
votes
1answer
129 views
Switchboard queries with NXT (the TEXT function)
The NXT query language includes the query function TEXT, which should return the textual content of the variable it applies to. In theory, Switchboard should support this. The file README.SWBD-QUERIES....
1
vote
2answers
3k views
Most commonly used words across languages?
I'm doing my own (very small) corpus analysis, including a word frequency list and would like to compare the top, say, 30 words with other such lists. I know that there are things like the GSL/NGSL, ...
2
votes
0answers
46 views
Is it possible to infer the meaning of a word of a language based on corpus analysis, without prior knowledge of the language?
If I am totally foreign to a language, are there corpus analysis methodologies and theories that I can employ to figure out the meaning of a word in a corpus based on that language?
If yes, do point ...
7
votes
1answer
712 views
How can I calculate if the difference between two word frequencies in one corpus is significant?
I want to study orthographical variants, for example:
Can firefighter and fire-fighter be considered orthographical variants (i.e. the difference in frequency in a corpus is not statistically ...
1
vote
2answers
92 views
How can I search a corpus for easy sentences?
I would like to search a large corpus for example sentences which contain exclusively or a high proportion of words from the N most common words in some frequency dictionary (the language of the ...
0
votes
2answers
44 views
Are there corpora which tag phrases by discourse- or conversational-function?
More generally, imagine a corpus which could identify the conversational function of a sentence, such as, "Have a nice day!", in narrow categories like, "ending a conversation", or broad ones like "...
0
votes
3answers
66 views
Which is the most recommended corpus for training a POS Tagger?
The tagger is for English and I will use the universal dependency tag set. I am looking for a corpus that best represents nowadays English.
1
vote
1answer
31 views
Praat - window size larger than vowel
I'm doing an acoustic-phonetic analysis of read-speech in an American English corpus.
If we want to set a window/frame size (25 - 30 ms) to cover the majority of vowels in the corpus, but some vowels (...
2
votes
0answers
41 views
Word alignment for Switchboard Dialogue Acts Corpus
I am trying to connect the dialogue acts of the Switchboard Dialogue Acts Corpus with the word alignment timing information available here.
I can see how it is possible to match up every utterance ...
2
votes
0answers
108 views
Is the source list of “Oxford English Corpus” public available?
The Oxford English Corpus (https://en.wikipedia.org/wiki/Oxford_English_Corpus) is a text corpus of 21st century English, used by the makers of the Oxford English Dictionary and by Oxford University ...
4
votes
3answers
1k views
Word frequency list for agglutinative languages like Swahili?
I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
2
votes
0answers
85 views
What word category do twink, adonis, muse etc. belong to? [closed]
Terms such as candy, cutie, honey, princess, diamond, queen(?), stud and bunny are terms of endearment (these terms are often used in a relationship to show affection and may also be used, with ...
1
vote
1answer
60 views
Spoken Arabic dictionary/corpus?
Is there an existing corpus or dictionary for different spoken Arabic dialects? I'd like to look up a word (in English or MSA or in a spoken dialect) and find where this word or phrase is spoken (and ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
6
votes
1answer
54 views
Corpus of English sentences written by non-native speakers?
I am working on a hobby project which tends to help the non-native speakers figure out if the sentences they write are good enough and sounds like written by English native speakers. Any relevant info ...
1
vote
1answer
55 views
Comparing corpora
So first, a disclaimer: I'm less than beginner level in everything that has to do with programming.
I'm analysing the occurrences of English code-switching/translanguaging in a particular Polish-...
3
votes
0answers
43 views
Searching for an English Whats-app Corpus
I am searching an English Whats-app Corpus in order to analyse a linguistic phenomena. I had some difficulties to find one and maybe some of you can help me out. It is only for corpus driven study and ...
1
vote
0answers
40 views
Where are some of the biggest freely available English 1 on 1 conversation corpuses that are in plain text preferably?
I'm looking for a free, preferably plain text, 1 on 1 English casual conversation (such as texting back and forth) corpus that requires little to no preprocessing (finding and replacing characters). ...
6
votes
1answer
104 views
How to make a Corpus? (selecting a good structure)
I'm making a Corpus.
I have searched for good practices on how to build it.
I've downloaded some Corpus to check their internal structure and the extension file, but I didn't find many.
So, I am lost ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
1
vote
4answers
206 views
Is it possible to build thesaurus automatically?
Is it possible to build thesaurus by simply analyzing lots of text, or it's obligatory should be handmade?
Also a question: is there free-to-download&use English language thesaurus?
1
vote
1answer
46 views
Suggestion for software to analyze data from social media- please advise
I am searching urgent for a software that I can use to analyse corpora. In short, I would like to have a look at collocations, ellipsis, frequencies, concordances etc. I also want to search for themes ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
4
votes
0answers
473 views
Parallel English-Spanish corpora?
I would like to know of parallel corpora for English-Spanish (that is, texts or language originally in English translated into Spanish and aligned by excerpts of texts) other than those on the Opus ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
5
votes
1answer
112 views
What are the conventions for corpus usage?
Suppose I'm working with a language that doesn't have much data available; the largest corpus around is a translation of the Bible, which is free to read online, but is presumably still copyrighted by ...
4
votes
2answers
323 views
JSON for corpora
Do you know any cases of corpus linguistics tools or corpora, using JSON to "mark" or delimited linguistics units in morphology, syntax,semantics? It seems all marking systems use some kind of XML or ...
1
vote
1answer
49 views
Is there a comprehensive list of corpora of all the languages (maybe even dialects) of the world?
Is there a comprehensive list of corpora of all the languages of the world?
I've been searching on wikipedia eg. https://en.wikipedia.org/wiki/Czech_National_Corpus. If I scroll to the bottom, I do ...
jk - Reinstate Monica
20.3k3 gold badges44 silver badges89 bronze badges
11
votes
3answers
3k views
Where to find frequency list of English words from newspapers, books and magazines?
I'd like to download a frequency list of English words that are most common in newspapers, books and magazines. It should contain at least 50.000 words.
Is it possible to download such a list from ...
2
votes
1answer
129 views
Does anyone have a link to the American Local News Corpus V1.0?
I found a paper http://www.cs.jhu.edu/~anni/papers/alnc_lrec14.pdf that has made this epic text corpus of over 1 billion words available somewhere, but I can't find it anywhere online.
The paper says ...
0
votes
1answer
32 views
Need an online freely available Anaphorically Annotated Corpus of English Language for Identification of Discourse Units
I'm in need of a freely available corpus of English language (which may contain text of any genre) but has to be annotated anaphoriacally i-e. Anaphora resolution should have been performed already.
...
3
votes
0answers
58 views
Is there an english news corpus available to download for between 1900 and 201X (free or low cost)
I'm attempting a word embedding analysis (think underlying meaning and implications, but computational) of certain keywords through time in the English language, but I am having some difficulty ...
1
vote
1answer
26 views
Is there a way to search for language data by context?
For example, if I wanted to see real-life transcriptions of dialogue in situations like a date, a business meeting, two friends walking in a park, etc, just to be able to generate lists of particular ...
3
votes
1answer
335 views
Different discounting methods with SRILM toolikt
I want to do some smoothing on a corpus and compare the results.
for train language model without smoothing I used "-addsmooth 0" option
now I should add two different smoothing and compare them, I ...
3
votes
1answer
48 views
Are there any corpora of informal and unstructured text labelled for Named Entity Recognition?
I have been searching since last week for annotated informal texts (with a lot of misspelled words, slang, etc.) to test some Named Entity Recognition tools for research purposes. For example, it ...
2
votes
3answers
198 views
Extensive English word corpora
I am looking for an extensive list of english words[including American and British... just an exhaustive list]. This list of english words should contain an exhaustive collection of all forms of all ...
2
votes
0answers
262 views
Collocation: interpreting contingency table for log-likelihood measure
I have a conundrum here.
I'm attempting to recreate the log-likelihood test example in 'The Cambridge Handbook of Corpus Linguistics (2015). However when I run the same test on the same data I get a ...
