Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
2
votes
1answer
151 views

What is considered the smallest possible sample size for word frequency lists used in FL instruction?

I've been engaged in a conversation on another site pertaining to frequency analysis, particularly in relation to the 1966 work Buntús Gaeilge, Colmán Ó Huallacháin; Ireland. Department of ...
1
vote
1answer
288 views

What are lexical and morpho-syntactic alternations?

I ought to write a paper with corpus-based analysis of a lexical or morpho-syntactic alternation. In other words, the paper should deal with two (or more) nearly synonymous lexical items or morpho-...
0
votes
1answer
129 views

Switchboard queries with NXT (the TEXT function)

The NXT query language includes the query function TEXT, which should return the textual content of the variable it applies to. In theory, Switchboard should support this. The file README.SWBD-QUERIES....
3
votes
2answers
366 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
2
votes
0answers
63 views

What are the subsets of the Eijiro corpus about?

I'm working with 2 japanese corpora called 'waei' and 'reiji'. They are legacy data, so I have no README or else to tell me the topic of either of them. But I need to know what they are about. My ...
4
votes
0answers
547 views

Is there any corpus for idioms?

I'm looking for a corpus for English (American, GB, Australian) idioms. Preferably created manually, because I already have two, but they are rather small and were built semi-automatically.
4
votes
1answer
1k views

Comparing frequencies of two corpora

I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
4
votes
1answer
154 views

Is there a web-based corpus tools that I can upload and use with my own corpus?

I am looking for a ready-made concordancer or collocator with a simple search interface that I can upload on my website and have my students use to search a custom corpus. I have been working with ...
9
votes
1answer
10k views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
2
votes
0answers
377 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
14
votes
4answers
5k views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
5
votes
1answer
336 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
0
votes
0answers
205 views

Tabulated lists of examples

"Everybody [?] knows" that there are these pairs of corresponding words in German and English in which an "f" appears in German where a "p" appears in English: Bischoff, ...
6
votes
3answers
2k views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
3
votes
1answer
126 views

Corpus analysis program

I'm looking for a program that will count information such as the most common terms in text, number of pronouns etc. I'm looking to input many song lyrics to see the most prevalent terms and the usage ...
5
votes
2answers
784 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
3
votes
0answers
203 views

Where can I find Japanese-English (manually) word-aligned corpora?

I'm looking for as many very reliable Japanese-English corpora as possible so I'd like to ask: Are there any other manually aligned corpora besides KFFT's and Utiyama's? What are the most accurate ...
2
votes
0answers
134 views

Semi-automatic annotation tool like '@nnotate'

Is there an annotation tool for linguistic structure that compares to @nnotate? One of the major features of annotate is the annotation is semi-automatic, i.e., it uses a language model to make ...
7
votes
1answer
186 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
3
votes
2answers
99 views

Symbols that compose a word

Words contain letters (A to Z and a to z), but also apostrophes, such as in the case: don't and even: 'em and even hyphens in the case of: ice-cream or: long wind- ed sentence With regard to ...
1
vote
1answer
172 views

How is the dependency of corpus size and observed ngrams?

I would like to know how the relation of corpus size to observed ngrams is(especially unigrams). Since the word frequency follows a Zipf distribution, i would expect some kind of logarithmic relation ...
6
votes
2answers
3k views

How to establish word frequencies from a corpus?

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
-1
votes
1answer
157 views

What factors should I consider when setting a Linguistic Corpus price?

What are the factors that must be considered when setting a linguistic corpus price? Is there any standard model or something else I should follow?
2
votes
1answer
141 views

Annotation agreement for multiple category assignment

There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs. e.g: the same coder c1 assigns the two categories (v1,v2) ...
25
votes
2answers
45k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
4
votes
1answer
174 views

Word list sought based on corpus of 19th century scientific English

I am currently working on an M.Phil thesis which is focusing on topic modelling 19th century journals. The journals in question are science, literature and antiquities based. I have extracted the ...
13
votes
3answers
522 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
3
votes
3answers
3k views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
2
votes
0answers
54 views

Generalisations which a bi-gram probabilistic model might infer from a dataset

I have the following exam question for a machine translation course: From my understanding, I assume the answer is looking for incorrect English grammar which get discovered by bi-grams. So the ...
5
votes
1answer
408 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
4
votes
1answer
111 views

Automated methods to align text

I have an assignment for my machine translation class which involves using the different language versions of websites as parallel corpora. The problem is the text isn't properly aligned. For example:...
5
votes
2answers
175 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
2
votes
1answer
157 views

Do generative linguists use spoken-word corpora?

Do generative linguists use spoken-word corpora for data? Offhand, I don't see why at least some of them wouldn't. I'm not suggesting that the use of spoken-word corpora vs. documented native ...
5
votes
1answer
804 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
9
votes
3answers
1k views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...
3
votes
2answers
335 views

Getting familiar with accents

Would it not be nice to have a site from which people can listen to different phrases in different languages with each phrase having the following characteristics, in any combination: Sample Sentence ...
4
votes
1answer
76 views

Database of synonym gradients

I'm going to be analyzing tweets where I care about how confidently someone is expressing an idea. So: X destroyed Y Would be express greater confidence than X beat Y About 9 months ago I ...
4
votes
1answer
200 views

A classical text about the Sun?

I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
7
votes
1answer
226 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...
3
votes
0answers
373 views

What is the formula for Usage Rate?

I read about a concept called the "usage rate" (proposed by Juliand and Chang-Rodriguez). It's a method for calculating the frequency of a word in a corpus. What exactly is the "usage rate" and what ...
9
votes
1answer
393 views

What is the Relationship Between Document Length and Unique Words

Sorry if my question is a little more mathematical in nature, but my question is: Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
4
votes
0answers
500 views

Tool for manually POS tagging texts

I'm interested if there is a text or set of texts where each word is correctly POS tagged. I know there are algorithms that can associate POS tags to the words, but there are always many of ...
4
votes
1answer
84 views

Corpus-making guide in Language Understanding Context

I want to make a standard annotated corpora on Language Understanding but I don't know where to start and what to do! Does anyone know any good reference in order to guide me through making a standard ...
10
votes
1answer
566 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
4
votes
1answer
2k views

Is there a computational method to syllabify English words?

There are straightforward ways to convert English words to phonemes via a dictionary that contains such information. However, is there a way to automatically convert English text into syllables? I.e., ...
3
votes
1answer
275 views

Dimensional reduction with synonomy and polysemy

I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single ...
4
votes
3answers
1k views

Word frequency list for agglutinative languages like Swahili?

I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
11
votes
3answers
3k views

Where to find frequency list of English words from newspapers, books and magazines?

I'd like to download a frequency list of English words that are most common in newspapers, books and magazines. It should contain at least 50.000 words. Is it possible to download such a list from ...
2
votes
2answers
52 views

Query format for NP without subject in PPCEME

As I, not being very familiar with corpus linguistics, read the manual for searching through PPCEME I entered: node: NP* print_indices: true query: (NP-SBJ *PRO*) without getting a result. Could ...
9
votes
1answer
229 views

Is there an automatic way of identifying transitive verbs in Computational Linguistics?

Is there any straightforward way of identifying transitive verbs (or sentences containing transitive constructions) in an BrE English text? I've looked into semantic shallow parsers, such as Semafor, ...