A database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

learn more… | top users | synonyms (1)

0
votes
1answer
4 views

How to incorporate sentences into a new work without infringing the copyright of anyone?

If you are developing a dictionary, or flash cards, or any other linguistics product that would need lots of real-life examples, and publish them (not just draw conclusions based on them), how can you ...
1
vote
0answers
25 views

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Alternatively, do any larger corpora have categories that pertain to technology, computer science, etc.? Or collections of English documentation and/or manuals related to modern devices, interfaces, ...
-1
votes
0answers
15 views

NLP questions to ask about a Corpus [on hold]

I am taking a Linguistics class and need to ask 2 questions about my corpus and then find the answers by parsing, text analysis, and NLP sites. The text is not clear on what questions should/could be ...
2
votes
1answer
55 views

Word difficulty as a function of frequency

What is known in general about the relationship between word difficulty and word frequency? That is, for a given word, suppose I know the frequency with which it (or its word stem) is encountered in ...
5
votes
1answer
63 views

How should grammatical errors be treated in parsing and tokenisation?

I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are ...
0
votes
0answers
30 views

German sentence corpus with cloze norms

I wish to conduct research on German based on cloze probabilities. Cloze probabilities are not trivial to collect, so I am asking if there exist any publicly available corpora of German sentences with ...
5
votes
1answer
98 views

copyright issues when making a corpus from the web

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I ...
1
vote
1answer
60 views

Time annotated corpus: plain text english corpus

I am working on how entities take a new sense over time. I am trying to find out a large english corpus (free to download) which should have the time annotation of the origin of the text. I suppose ...
0
votes
0answers
41 views

How to annotate a corpus of written discourse for pragmatic analysis (speech acts)?

I am currently creating a small-ish corpus of online conversations that I wish to analyze quantitatively and qualitatively from a pragmatic perspective (speech acts level mainly). The problem is that ...
1
vote
0answers
51 views

Is there a data set of elementary typical phrases translated in different languages?

To be more explicit, I think about a collection of "abstract" sentences which could be categorized in such a way that they would be easily identified in any language. For example, some typical phrases ...
2
votes
1answer
40 views

What is considered the smallest possible sample size for word frequency lists used in FL instruction?

I've been engaged in a conversation on another site pertaining to frequency analysis, particularly in relation to the 1966 work Buntús Gaeilge, Colmán Ó Huallacháin; Ireland. Department of ...
0
votes
0answers
32 views

What are lexical and morpho-syntactic alternations?

I ought to write a paper with corpus-based analysis of a lexical or morpho-syntactic alternation. In other words, the paper should deal with two (or more) nearly synonymous lexical items or ...
0
votes
0answers
37 views

Switchboard queries with NXT (the TEXT function)

The NXT query language includes the query function TEXT, which should return the textual content of the variable it applies to. In theory, Switchboard should support this. The file ...
2
votes
1answer
41 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
2
votes
0answers
43 views

What are the subsets of the Eijiro corpus about?

I'm working with 2 japanese corpora called 'waei' and 'reiji'. They are legacy data, so I have no README or else to tell me the topic of either of them. But I need to know what they are about. My ...
3
votes
0answers
73 views

Is there any corpus for idioms?

I'm looking for a corpus for English (American, GB, Australian) idioms. Preferably created manually, because I already have two, but they are rather small and were built semi-automatically.
2
votes
1answer
79 views

Comparing frequencies of two corpora

I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
2
votes
1answer
56 views

Is there a web-based corpus tools that I can upload and use with my own corpus?

I am looking for a ready-made concordancer or collocator with a simple search interface that I can upload on my website and have my students use to search a custom corpus. I have been working with ...
1
vote
1answer
158 views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
0
votes
0answers
34 views

Free animal corpus for GWAP

I'm working on this game: http://nlp.fi.muni.cz/projekty/qgame/ The goal is to play the game to build a knowledge base. For example, a database similar to Concept Net. The game can only guess ...
1
vote
0answers
111 views

I am looking for an Arabic ngram corpus

I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages ...
6
votes
3answers
504 views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
5
votes
1answer
168 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
0
votes
0answers
52 views

Tabulated lists of examples

"Everybody [?] knows" that there are these pairs of corresponding words in German and English in which an "f" appears in German where a "p" appears in English: Bischoff, bishop Schiff, ship ...
3
votes
0answers
118 views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
3
votes
1answer
92 views

Corpus analysis program

I'm looking for a program that will count information such as the most common terms in text, number of pronouns etc. I'm looking to input many song lyrics to see the most prevalent terms and the usage ...
0
votes
0answers
69 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
3
votes
0answers
83 views

Where can I find Japanese-English (manually) word-aligned corpora?

I'm looking for as many very reliable Japanese-English corpora as possible so I'd like to ask: Are there any other manually aligned corpora besides KFFT's and Utiyama's? What are the most accurate ...
2
votes
0answers
41 views

Semi-automatic annotation tool like '@nnotate'

Is there an annotation tool for linguistic structure that compares to @nnotate? One of the major features of annotate is the annotation is semi-automatic, i.e., it uses a language model to make ...
2
votes
1answer
76 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
3
votes
2answers
75 views

Symbols that compose a word

Words contain letters (A to Z and a to z), but also apostrophes, such as in the case: don't and even: 'em and even hyphens in the case of: ice-cream or: long wind- ed sentence With regard ...
1
vote
1answer
77 views

How is the dependency of corpus size and observed ngrams?

I would like to know how the relation of corpus size to observed ngrams is(especially unigrams). Since the word frequency follows a Zipf distribution, i would expect some kind of logarithmic relation ...
4
votes
1answer
345 views

Establishing word frequencies from a corpus

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
-1
votes
1answer
109 views

What factors should I consider when setting a Linguistic Corpus price?

What are the factors that must be considered when setting a linguistic corpus price? Is there any standard model or something else I should follow?
2
votes
1answer
49 views

Annotation agreement for multiple category assignment

There are annotation tasks where the items belong to multiple categories and annotators have to mark each category to which the item belongs. e.g: the same coder c1 assigns the two categories (v1,v2) ...
2
votes
2answers
4k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
2
votes
1answer
123 views

Word list sought based on corpus of 19th century scientific English

I am currently working on an M.Phil thesis which is focusing on topic modelling 19th century journals. The journals in question are science, literature and antiquities based. I have extracted the ...
7
votes
1answer
184 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
2
votes
3answers
460 views

Online bigram frequency lookup

For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
2
votes
0answers
40 views

Generalisations which a bi-gram probabilistic model might infer from a dataset

I have the following exam question for a machine translation course: From my understanding, I assume the answer is looking for incorrect English grammar which get discovered by bi-grams. So the ...
3
votes
1answer
131 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
4
votes
1answer
87 views

Automated methods to align text

I have an assignment for my machine translation class which involves using the different language versions of websites as parallel corpora. The problem is the text isn't properly aligned. For ...
4
votes
2answers
109 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
2
votes
1answer
118 views

Do generative linguists use spoken-word corpora?

Do generative linguists use spoken-word corpora for data? Offhand, I don't see why at least some of them wouldn't. I'm not suggesting that the use of spoken-word corpora vs. documented native ...
5
votes
1answer
342 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
7
votes
3answers
427 views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...
3
votes
2answers
160 views

Getting familiar with accents

Would it not be nice to have a site from which people can listen to different phrases in different languages with each phrase having the following characteristics, in any combination: Sample ...
4
votes
1answer
58 views

Database of synonym gradients

I'm going to be analyzing tweets where I care about how confidently someone is expressing an idea. So: X destroyed Y Would be express greater confidence than X beat Y About 9 months ago ...
3
votes
1answer
116 views

A classical text about the Sun?

I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
5
votes
1answer
81 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...