A database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

learn more… | top users | synonyms (1)

0
votes
0answers
18 views

Is there a data set of elementary typical phrases translated in different languages?

To be more explicit, I think about a collection of "abstract" sentences which could be categorized in such a way that they would be easily identified in any language. For example, some typical phrases ...
2
votes
1answer
28 views

What is considered the smallest possible sample size for word frequency lists used in FL instruction?

I've been engaged in a conversation on another site pertaining to frequency analysis, particularly in relation to the 1966 work Buntús Gaeilge, Colmán Ó Huallacháin; Ireland. Department of ...
0
votes
0answers
24 views

What are lexical and morpho-syntactic alternations?

I ought to write a paper with corpus-based analysis of a lexical or morpho-syntactic alternation. In other words, the paper should deal with two (or more) nearly synonymous lexical items or ...
0
votes
0answers
33 views

Switchboard queries with NXT (the TEXT function)

The NXT query language includes the query function TEXT, which should return the textual content of the variable it applies to. In theory, Switchboard should support this. The file ...
2
votes
1answer
31 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
2
votes
0answers
37 views

What are the subsets of the Eijiro corpus about?

I'm working with 2 japanese corpora called 'waei' and 'reiji'. They are legacy data, so I have no README or else to tell me the topic of either of them. But I need to know what they are about. My ...
3
votes
0answers
65 views

Is there any corpus for idioms?

I'm looking for a corpus for English (American, GB, Australian) idioms. Preferably created manually, because I already have two, but they are rather small and were built semi-automatically.
2
votes
1answer
59 views

Comparing frequencies of two corpora

I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
2
votes
1answer
52 views

Is there a web-based corpus tools that I can upload and use with my own corpus?

I am looking for a ready-made concordancer or collocator with a simple search interface that I can upload on my website and have my students use to search a custom corpus. I have been working with ...
1
vote
1answer
73 views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
0
votes
0answers
31 views

Free animal corpus for GWAP

I'm working on this game: http://nlp.fi.muni.cz/projekty/qgame/ The goal is to play the game to build a knowledge base. For example, a database similar to Concept Net. The game can only guess ...
1
vote
0answers
92 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
6
votes
3answers
379 views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
5
votes
1answer
160 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
0
votes
0answers
50 views

Tabulated lists of examples

"Everybody [?] knows" that there are these pairs of corresponding words in German and English in which an "f" appears in German where a "p" appears in English: Bischoff, bishop Schiff, ship ...
2
votes
0answers
82 views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
3
votes
1answer
88 views

Corpus analysis program

I'm looking for a program that will count information such as the most common terms in text, number of pronouns etc. I'm looking to input many song lyrics to see the most prevalent terms and the usage ...
0
votes
0answers
56 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
3
votes
0answers
77 views

Where can I find Japanese-English (manually) word-aligned corpora?

I'm looking for as many very reliable Japanese-English corpora as possible so I'd like to ask: Are there any other manually aligned corpora besides KFFT's and Utiyama's? What are the most accurate ...
2
votes
0answers
40 views

Semi-automatic annotation tool like '@nnotate'

Is there an annotation tool for linguistic structure that compares to @nnotate? One of the major features of annotate is the annotation is semi-automatic, i.e., it uses a language model to make ...
2
votes
1answer
72 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
3
votes
2answers
74 views

Symbols that compose a word

Words contain letters (A to Z and a to z), but also apostrophes, such as in the case: don't and even: 'em and even hyphens in the case of: ice-cream or: long wind- ed sentence With regard ...
1
vote
1answer
64 views

How is the dependency of corpus size and observed ngrams?

I would like to know how the relation of corpus size to observed ngrams is(especially unigrams). Since the word frequency follows a Zipf distribution, i would expect some kind of logarithmic relation ...
4
votes
1answer
263 views

Establishing word frequencies from a corpus

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
-1
votes
1answer
105 views

What factors should I consider when setting a Linguistic Corpus price?

What are the factors that must be considered when setting a linguistic corpus price? Is there any standard model or something else I should follow?
2
votes
1answer
46 views

Annotation agreement for multiple category assignment

There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs. e.g: the same coder c1 assigns the two categories (v1,v2) ...
1
vote
2answers
2k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
2
votes
1answer
120 views

Word list sought based on corpus of 19th century scientific English

I am currently working on an M.Phil thesis which is focusing on topic modelling 19th century journals. The journals in question are science, literature and antiquities based. I have extracted the ...
7
votes
1answer
152 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
2
votes
3answers
331 views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
2
votes
0answers
39 views

Generalisations which a bi-gram probabilistic model might infer from a dataset

I have the following exam question for a machine translation course: From my understanding, I assume the answer is looking for incorrect English grammar which get discovered by bi-grams. So the ...
3
votes
1answer
113 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
4
votes
1answer
87 views

Automated methods to align text

I have an assignment for my machine translation class which involves using the different language versions of websites as parallel corpora. The problem is the text isn't properly aligned. For ...
4
votes
2answers
105 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
2
votes
1answer
115 views

Do generative linguists use spoken-word corpora?

Do generative linguists use spoken-word corpora for data? Offhand, I don't see why at least some of them wouldn't. I'm not suggesting that the use of spoken-word corpora vs. documented native ...
5
votes
1answer
283 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
7
votes
3answers
354 views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...
3
votes
2answers
151 views

Getting familiar with accents

Would it not be nice to have a site from which people can listen to different phrases in different languages with each phrase having the following characteristics, in any combination: Sample ...
4
votes
1answer
58 views

Database of synonym gradients

I'm going to be analyzing tweets where I care about how confidently someone is expressing an idea. So: X destroyed Y Would be express greater confidence than X beat Y About 9 months ago ...
3
votes
1answer
113 views

A classical text about the Sun?

I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
5
votes
1answer
77 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...
1
vote
0answers
143 views

What is the formula for Usage Rate?

I read about a concept called the "usage rate" (proposed by Juliand and Chang-Rodriguez). It's a method for calculating the frequency of a word in a corpus. What exactly is the "usage rate" and what ...
8
votes
1answer
203 views

What is the Relationship Between Document Length and Unique Words

Sorry if my question is a little more mathematical in nature, but my question is: Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
4
votes
0answers
127 views

Tool for manually POS tagging texts

I'm interested if there is a text or set of texts where each word is correctly POS tagged. I know there are algorithms that can associate POS tags to the words, but there are always many of ...
3
votes
1answer
60 views

Corpus-making guide in Language Understanding Context

I want to make a standard annotated corpora on Language Understanding but I don't know where to start and what to do! Does anyone know any good reference in order to guide me through making a standard ...
9
votes
1answer
270 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
0
votes
0answers
626 views

Is there a computational method to syllabify English words?

There are straightforward ways to convert English words to phonemes via a dictionary that contains such information. However, is there a way to automatically convert English text into syllables? I.e., ...
3
votes
1answer
140 views

Dimensional reduction with synonomy and polysemy

I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single ...
3
votes
1answer
358 views

Word frequency list for agglutinative languages like Swahili?

I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
3
votes
1answer
2k views

Most common english words from newspapers, books and magazines

I'd like to download a frequency list of english words that are most common in newspapers, books and magazines. It should contain at least 50 000 words. Is it possible to download such a list from ...