Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
0
votes
2answers
301 views

Do we have a taxonomy other than part of speech?

I'm trying to create an educational system, and I would like to have more taxonomies than a simple part of speech. For example, I would like to be able to categorize words into: Tree => object ...
2
votes
2answers
83 views

What are some canonical or seminal corpus linguistic studies using the Brown Corpus?

I'm looking for some good works of corpus linguistics using the Brown Corpus. Does anyone know of an important early work, and/or an important recent work that uses it?
0
votes
0answers
36 views

Phrase suggestion model using a gold standard corpus

There are already spell checking models available which help us to find the suggested correct spellings based on a corpus of trained correct spellings. Can the granularity be increased to "word" from ...
4
votes
1answer
648 views

Question type corpus

I'm looking for a corpus that contains questions and question types. Like in this article, but they didn't attach any link to their corpus. http://polibits.gelbukh.com/2009_40/40_01.pdf
1
vote
0answers
82 views

Does corpus linguistics research confirm/disconfirm Bolinger principle?

I remember learning about Bolinger's principle (basically, when there is a verb complement situation like "I like [verb complement]" or "I want to [verb complement]", gerunds are preferred to express ...
1
vote
1answer
65 views

How do you search for nouns that contain the sequence of letters sb in the Corpus of Contemporary American English?

I am trying to design a word list for a phonetics project. I can use wildcard searches in the corpus such as * sb * which will return all words that contain this sequence of letters (either medially ...
4
votes
1answer
111 views

Automated methods to align text

I have an assignment for my machine translation class which involves using the different language versions of websites as parallel corpora. The problem is the text isn't properly aligned. For example:...
0
votes
1answer
108 views

Reason for using 4- and 5-grams for co-occurrence analysis

I am trying to replicate the results of a paper which didn't share its data methods that clearly. It makes use of the Web 1T 5-gram Version 1 data set collected from Google to expand 72 words into 964 ...
2
votes
1answer
283 views

What are lexical and morpho-syntactic alternations?

I ought to write a paper with corpus-based analysis of a lexical or morpho-syntactic alternation. In other words, the paper should deal with two (or more) nearly synonymous lexical items or morpho-...
1
vote
0answers
44 views

Researching extraposition in a syntactic treebank

I'm writing a paper on extraposition in English (and other right-branching discontinuities). I have found a lot of interesting theory on this but the instructions say that if possible concrete corpus ...
1
vote
1answer
265 views

Evaluate idea to autobuild russian-english parallel corpus

I was looking for russian-english parallel corpus, but have found none publicly available to the moment. If you know one, or parallel text database, I would be very grateful for one. (I heard of ...
3
votes
2answers
364 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
0
votes
2answers
129 views

Expected Frequency

I am reading an article where the authors mention something they call the expected frequency of a word in a corpus without explaining or giving a definition of what this means. The article in ...
4
votes
1answer
2k views

Is there a computational method to syllabify English words?

There are straightforward ways to convert English words to phonemes via a dictionary that contains such information. However, is there a way to automatically convert English text into syllables? I.e., ...
5
votes
1answer
101 views

How should grammatical errors be treated in parsing and tokenisation?

I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are ...
0
votes
0answers
132 views

How to annotate a corpus of written discourse for pragmatic analysis (speech acts)?

I am currently creating a small-ish corpus of online conversations that I wish to analyze quantitatively and qualitatively from a pragmatic perspective (speech acts level mainly). The problem is that ...
1
vote
0answers
62 views

Is there a data set of elementary typical phrases translated in different languages?

To be more explicit, I think about a collection of "abstract" sentences which could be categorized in such a way that they would be easily identified in any language. For example, some typical phrases ...
2
votes
1answer
150 views

What is considered the smallest possible sample size for word frequency lists used in FL instruction?

I've been engaged in a conversation on another site pertaining to frequency analysis, particularly in relation to the 1966 work Buntús Gaeilge, Colmán Ó Huallacháin; Ireland. Department of ...
2
votes
0answers
63 views

What are the subsets of the Eijiro corpus about?

I'm working with 2 japanese corpora called 'waei' and 'reiji'. They are legacy data, so I have no README or else to tell me the topic of either of them. But I need to know what they are about. My ...
4
votes
0answers
547 views

Is there any corpus for idioms?

I'm looking for a corpus for English (American, GB, Australian) idioms. Preferably created manually, because I already have two, but they are rather small and were built semi-automatically.
4
votes
1answer
1k views

Comparing frequencies of two corpora

I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
4
votes
1answer
151 views

Is there a web-based corpus tools that I can upload and use with my own corpus?

I am looking for a ready-made concordancer or collocator with a simple search interface that I can upload on my website and have my students use to search a custom corpus. I have been working with ...
25
votes
2answers
44k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
2
votes
0answers
374 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
5
votes
1answer
332 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
7
votes
3answers
949 views

Are there dictionaries like Collins COBUILD for other languages than English? [closed]

Let me explain what I'm after: monolingual all examples are taken from a large corpus of contemporary text (the COBUILD series is now using Bank of English) the senses for each entry are sorted by ...
0
votes
0answers
205 views

Tabulated lists of examples

"Everybody [?] knows" that there are these pairs of corresponding words in German and English in which an "f" appears in German where a "p" appears in English: Bischoff, ...
3
votes
1answer
126 views

Corpus analysis program

I'm looking for a program that will count information such as the most common terms in text, number of pronouns etc. I'm looking to input many song lyrics to see the most prevalent terms and the usage ...
3
votes
0answers
203 views

Where can I find Japanese-English (manually) word-aligned corpora?

I'm looking for as many very reliable Japanese-English corpora as possible so I'd like to ask: Are there any other manually aligned corpora besides KFFT's and Utiyama's? What are the most accurate ...
2
votes
0answers
133 views

Semi-automatic annotation tool like '@nnotate'

Is there an annotation tool for linguistic structure that compares to @nnotate? One of the major features of annotate is the annotation is semi-automatic, i.e., it uses a language model to make ...
9
votes
3answers
2k views

How to work on annotating AND sentence-aligning parallel texts?

There are plenty of software programs facilitating interlinear transcription (e.g. Toolbox, Fieldworks Language Explorer). There's also a number of tools that allow you to work on aligning parallel ...
7
votes
1answer
183 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
3
votes
2answers
99 views

Symbols that compose a word

Words contain letters (A to Z and a to z), but also apostrophes, such as in the case: don't and even: 'em and even hyphens in the case of: ice-cream or: long wind- ed sentence With regard to ...
1
vote
1answer
171 views

How is the dependency of corpus size and observed ngrams?

I would like to know how the relation of corpus size to observed ngrams is(especially unigrams). Since the word frequency follows a Zipf distribution, i would expect some kind of logarithmic relation ...
-1
votes
1answer
157 views

What factors should I consider when setting a Linguistic Corpus price?

What are the factors that must be considered when setting a linguistic corpus price? Is there any standard model or something else I should follow?
2
votes
1answer
155 views

Do generative linguists use spoken-word corpora?

Do generative linguists use spoken-word corpora for data? Offhand, I don't see why at least some of them wouldn't. I'm not suggesting that the use of spoken-word corpora vs. documented native ...
2
votes
1answer
138 views

Annotation agreement for multiple category assignment

There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs. e.g: the same coder c1 assigns the two categories (v1,v2) ...
3
votes
3answers
3k views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
4
votes
1answer
174 views

Word list sought based on corpus of 19th century scientific English

I am currently working on an M.Phil thesis which is focusing on topic modelling 19th century journals. The journals in question are science, literature and antiquities based. I have extracted the ...
5
votes
1answer
406 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
10
votes
1answer
560 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
2
votes
0answers
54 views

Generalisations which a bi-gram probabilistic model might infer from a dataset

I have the following exam question for a machine translation course: From my understanding, I assume the answer is looking for incorrect English grammar which get discovered by bi-grams. So the ...
5
votes
2answers
172 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
5
votes
1answer
795 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
3
votes
2answers
330 views

Getting familiar with accents

Would it not be nice to have a site from which people can listen to different phrases in different languages with each phrase having the following characteristics, in any combination: Sample Sentence ...
4
votes
1answer
76 views

Database of synonym gradients

I'm going to be analyzing tweets where I care about how confidently someone is expressing an idea. So: X destroyed Y Would be express greater confidence than X beat Y About 9 months ago I ...
7
votes
1answer
225 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...
4
votes
1answer
198 views

A classical text about the Sun?

I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
9
votes
1answer
381 views

What is the Relationship Between Document Length and Unique Words

Sorry if my question is a little more mathematical in nature, but my question is: Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
4
votes
0answers
500 views

Tool for manually POS tagging texts

I'm interested if there is a text or set of texts where each word is correctly POS tagged. I know there are algorithms that can associate POS tags to the words, but there are always many of ...