A database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

learn more… | top users | synonyms (1)

0
votes
0answers
26 views

Free animal corpus for GWAP

I'm working on this game: http://nlp.fi.muni.cz/projekty/qgame/ The goal is to play the game to build a knowledge base. For example, a database similar to Concept Net. The game can only guess ...
1
vote
0answers
74 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
6
votes
3answers
292 views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
5
votes
1answer
137 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
0
votes
0answers
47 views

Tabulated lists of examples

"Everybody [?] knows" that there are these pairs of corresponding words in German and English in which an "f" appears in German where a "p" appears in English: Bischoff, bishop Schiff, ship ...
0
votes
0answers
56 views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
3
votes
1answer
71 views

Corpus analysis program

I'm looking for a program that will count information such as the most common terms in text, number of pronouns etc. I'm looking to input many song lyrics to see the most prevalent terms and the usage ...
0
votes
0answers
42 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
3
votes
0answers
64 views

Where can I find Japanese-English (manually) word-aligned corpora?

I'm looking for as many very reliable Japanese-English corpora as possible so I'd like to ask: Are there any other manually aligned corpora besides KFFT's and Utiyama's? What are the most accurate ...
2
votes
0answers
37 views

Semi-automatic annotation tool like '@nnotate'

Is there an annotation tool for linguistic structure that compares to @nnotate? One of the major features of annotate is the annotation is semi-automatic, i.e., it uses a language model to make ...
1
vote
1answer
67 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
3
votes
2answers
72 views

Symbols that compose a word

Words contain letters (A to Z and a to z), but also apostrophes, such as in the case: don't and even: 'em and even hyphens in the case of: ice-cream or: long wind- ed sentence With regard ...
1
vote
1answer
58 views

How is the dependency of corpus size and observed ngrams?

I would like to know how the relation of corpus size to observed ngrams is(especially unigrams). Since the word frequency follows a Zipf distribution, i would expect some kind of logarithmic relation ...
4
votes
1answer
148 views

Establishing word frequencies from a corpus

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
-1
votes
1answer
101 views

What factors should I consider when setting a Linguistic Corpus price?

What are the factors that must be considered when setting a linguistic corpus price? Is there any standard model or something else I should follow?
2
votes
1answer
44 views

Annotation agreement for multiple category assignment

There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs. e.g: the same coder c1 assigns the two categories (v1,v2) ...
1
vote
2answers
1k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
2
votes
1answer
115 views

Word list sought based on corpus of 19th century scientific English

I am currently working on an M.Phil thesis which is focusing on topic modelling 19th century journals. The journals in question are science, literature and antiquities based. I have extracted the ...
5
votes
1answer
120 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
2
votes
3answers
213 views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
2
votes
0answers
38 views

Generalisations which a bi-gram probabilistic model might infer from a dataset

I have the following exam question for a machine translation course: From my understanding, I assume the answer is looking for incorrect English grammar which get discovered by bi-grams. So the ...
3
votes
1answer
89 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
4
votes
1answer
84 views

Automated methods to align text

I have an assignment for my machine translation class which involves using the different language versions of websites as parallel corpora. The problem is the text isn't properly aligned. For ...
4
votes
2answers
99 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
2
votes
1answer
107 views

Do generative linguists use spoken-word corpora?

Do generative linguists use spoken-word corpora for data? Offhand, I don't see why at least some of them wouldn't. I'm not suggesting that the use of spoken-word corpora vs. documented native ...
5
votes
1answer
246 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
7
votes
3answers
290 views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...
3
votes
2answers
149 views

Getting familiar with accents

Would it not be nice to have a site from which people can listen to different phrases in different languages with each phrase having the following characteristics, in any combination: Sample ...
4
votes
1answer
57 views

Database of synonym gradients

I'm going to be analyzing tweets where I care about how confidently someone is expressing an idea. So: X destroyed Y Would be express greater confidence than X beat Y About 9 months ago ...
3
votes
1answer
110 views

A classical text about the Sun?

I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
5
votes
1answer
72 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...
1
vote
0answers
111 views

What is the formula for Usage Rate?

I read about a concept called the "usage rate" (proposed by Juliand and Chang-Rodriguez). It's a method for calculating the frequency of a word in a corpus. What exactly is the "usage rate" and what ...
8
votes
1answer
189 views

What is the Relationship Between Document Length and Unique Words

Sorry if my question is a little more mathematical in nature, but my question is: Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
4
votes
0answers
110 views

Tool for manually POS tagging texts

I'm interested if there is a text or set of texts where each word is correctly POS tagged. I know there are algorithms that can associate POS tags to the words, but there are always many of ...
3
votes
1answer
59 views

Corpus-making guide in Language Understanding Context

I want to make a standard annotated corpora on Language Understanding but I don't know where to start and what to do! Does anyone know any good reference in order to guide me through making a standard ...
9
votes
1answer
245 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
0
votes
0answers
518 views

Is there a computational method to syllabify English words?

There are straightforward ways to convert English words to phonemes via a dictionary that contains such information. However, is there a way to automatically convert English text into syllables? I.e., ...
3
votes
1answer
126 views

Dimensional reduction with synonomy and polysemy

I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single ...
3
votes
1answer
287 views

Word frequency list for agglutinative languages like Swahili?

I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
3
votes
1answer
2k views

Most common english words from newspapers, books and magazines

I'd like to download a frequency list of english words that are most common in newspapers, books and magazines. It should contain at least 50 000 words. Is it possible to download such a list from ...
2
votes
2answers
42 views

Query format for NP without subject in PPCEME

As I, not being very familiar with corpus linguistics, read the manual for searching through PPCEME I entered: node: NP* print_indices: true query: (NP-SBJ *PRO*) without getting a result. Could ...
7
votes
1answer
176 views

Is there an automatic way of identifying transitive verbs in Computational Linguistics?

Is there any straightforward way of identifying transitive verbs (or sentences containing transitive constructions) in an BrE English text? I've looked into semantic shallow parsers, such as Semafor, ...
4
votes
1answer
161 views

Is there a standard corpus against which to benchmark mechanical parsers?

I was wondering if there is a published standard corpus of English sentences which are confounding for mechanical parsers. I am looking over sentences given here, and I have the following (very short) ...
1
vote
2answers
141 views

1TB word (1-gram) corpus

Where can I find a 1TB or larger 1-gram corpus? I am comparing certain structures for term-frequency calculation, but without a significantly large dataset I won't be able to show the non-theoretical ...
7
votes
1answer
715 views

Does anyone know of text message corpora?

I am looking for a large corpus of text messages. By large, I am hoping to have at least 15,000 text messages in my sample. I am fine with combining several smaller corpora into a larger corpus as I ...
7
votes
3answers
774 views

How to work on annotating AND sentence-aligning parallel texts?

There are plenty of software programs facilitating interlinear transcription (e.g. Toolbox, Fieldworks Language Explorer). There's also a number of tools that allow you to work on aligning parallel ...
4
votes
0answers
264 views

How to decrease CRF++ feature function set?

I have a problem with the CRF++ Package. CRF++ cannot handle large training parts-of-speech corpora (large tagset and large number of words). In fact, the number of feature functions automatically ...
10
votes
1answer
380 views

What are some resources that I can use to gather Twitter data for an NLP project?

I was going to use the Edinburgh Twitter Corpus that is referenced in this paper: The Edinburgh Twitter Corpus. But apparently Twitter has changed their Terms and the corpus is no longer available. ...
5
votes
3answers
386 views

Are there dictionaries like Collins COBUILD for other languages than English? [closed]

Let me explain what I'm after: monolingual all examples are taken from a large corpus of contemporary text (the COBUILD series is now using Bank of English) the senses for each entry are sorted by ...
15
votes
4answers
853 views

Corpus of baby-talk or motherese

Baby-talk or motherese is the language that parents tend to use when addressing preverbal or just-starting-to-speak children. What are the fundamental features or rules that define this subset of a ...