A database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.
4
votes
1answer
37 views
Best method for buidling a learner corpus for DDL
I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
0
votes
1answer
28 views
Online bigram frequency lookup
For a project I would like to look up the top bigrams containing the word "like". Is there such a service available online somewhere?
2
votes
0answers
32 views
Generalisations which a bi-gram probabilistic model might infer from a dataset
I have the following exam question for a machine translation course:
From my understanding, I assume the answer is looking for incorrect English grammar which get discovered by bi-grams.
So the ...
2
votes
1answer
47 views
Semantic frame representation of ATIS 3 corpora
Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform.
There are many tagging ...
3
votes
0answers
48 views
Automated methods to align text
I have an assignment for my machine translation class which involves using the different language versions of websites as parallel corpora. The problem is the text isn't properly aligned.
For ...
4
votes
2answers
88 views
Corpora of Indigenous American Languages?
Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
2
votes
0answers
42 views
Do generative linguists use spoken-word corpora?
Do generative linguists use spoken-word corpora for data? Offhand, I don't see why at least some of them wouldn't.
I'm not suggesting that the use of spoken-word corpora vs. documented native ...
0
votes
0answers
25 views
Dictionary lists without names [duplicate]
Are there comprehensive free dictionary lists without names in English and German, at best with word flexions?
I.e.: frank and bush would be in it, george, georgetown and washington wouldn't be.
The ...
5
votes
1answer
87 views
Words and phrases more likely in everyday speech
I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
7
votes
3answers
107 views
Tools to annotate (categorise) sentences from a sentence corpus
I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm.
I am looking for a (preferably web-based) tool that:
Allows me to input a list ...
3
votes
2answers
120 views
Getting familiar with accents
Would it not be nice to have a site from which people can listen to
different phrases in different languages with each phrase having the
following characteristics, in any combination:
Sample ...
4
votes
1answer
52 views
Database of synonym gradients
I'm going to be analyzing tweets where I care about how confidently someone is expressing an idea. So:
X destroyed Y
Would be express greater confidence than
X beat Y
About 9 months ago ...
3
votes
1answer
98 views
A classical text about the Sun?
I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
5
votes
1answer
50 views
Is there a digital corpus somewhere of pre-Latin Vietnamese text?
I like as a hobby to do text processing and analysis of CJKV text.
Japanese and Chinese is of course available in near infinite quantities.
Korean is a bit harder to find, but not impossible.
...
1
vote
0answers
33 views
What is the formula for Usage Rate?
I read about a concept called the "usage rate" (proposed by Juliand and Chang-Rodriguez). It's a method for calculating the frequency of a word in a corpus. What exactly is the "usage rate" and what ...
8
votes
1answer
147 views
What is the Relationship Between Document Length and Unique Words
Sorry if my question is a little more mathematical in nature, but my question is:
Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
3
votes
0answers
75 views
Tool for manually POS tagging texts
I'm interested if there is a text or set of texts where each word is correctly POS tagged.
I know there are algorithms that can associate POS tags to the words, but there are always many of ...
3
votes
1answer
56 views
Corpus-making guide in Language Understanding Context
I want to make a standard annotated corpora on Language Understanding but I don't know where to start and what to do!
Does anyone know any good reference in order to guide me through making a standard ...
8
votes
1answer
175 views
The power of trigram language models (2nd order Markov models)
Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
0
votes
0answers
273 views
Is there a computational method to syllabify English words?
There are straightforward ways to convert English words to phonemes via a dictionary that contains such information. However, is there a way to automatically convert English text into syllables? I.e., ...
3
votes
1answer
108 views
Dimensional reduction with synonomy and polysemy
I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single ...
3
votes
1answer
159 views
Word frequency list for agglutinative languages like Swahili?
I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
3
votes
1answer
1k views
Most common english words from newspapers, books and magazines
I'd like to download a frequency list of english words that are most common in newspapers, books and magazines. It should contain at least 50 000 words.
Is it possible to download such a list from ...
2
votes
2answers
39 views
Query format for NP without subject in PPCEME
As I, not being very familiar with corpus linguistics, read the manual for searching through PPCEME I entered:
node: NP*
print_indices: true
query: (NP-SBJ *PRO*)
without getting a result.
Could ...
7
votes
1answer
155 views
Is there an automatic way of identifying transitive verbs in Computational Linguistics?
Is there any straightforward way of identifying transitive verbs (or sentences containing transitive constructions) in an BrE English text?
I've looked into semantic shallow parsers, such as Semafor, ...
4
votes
1answer
143 views
Is there a standard corpus against which to benchmark mechanical parsers?
I was wondering if there is a published standard corpus of English sentences which are confounding for mechanical parsers. I am looking over sentences given here, and I have the following (very short) ...
1
vote
2answers
129 views
1TB word (1-gram) corpus
Where can I find a 1TB or larger 1-gram corpus?
I am comparing certain structures for term-frequency calculation, but without a significantly large dataset I won't be able to show the non-theoretical ...
7
votes
1answer
314 views
Does anyone know of text message corpora?
I am looking for a large corpus of text messages. By large, I am hoping to have at least 15,000 text messages in my sample. I am fine with combining several smaller corpora into a larger corpus as I ...
7
votes
2answers
448 views
How to work on annotating AND sentence-aligning parallel texts?
There are plenty of software programs facilitating interlinear transcription (e.g. Toolbox, Fieldworks Language Explorer). There's also a number of tools that allow you to work on aligning parallel ...
4
votes
0answers
217 views
How to decrease CRF++ feature function set?
I have a problem with the CRF++ Package. CRF++ cannot handle large training parts-of-speech corpora (large tagset and large number of words).
In fact, the number of feature functions automatically ...
10
votes
1answer
280 views
What are some resources that I can use to gather Twitter data for an NLP project?
I was going to use the Edinburgh Twitter Corpus that is referenced in this paper: The Edinburgh Twitter Corpus. But apparently Twitter has changed their Terms and the corpus is no longer available. ...
5
votes
3answers
265 views
Are there dictionaries like Collins COBUILD for other languages than English? [closed]
Let me explain what I'm after:
monolingual
all examples are taken from a large corpus of contemporary text (the COBUILD series is now using Bank of English)
the senses for each entry are sorted by ...
14
votes
4answers
697 views
Corpus of baby-talk or motherese
Baby-talk or motherese is the language that parents tend to use when addressing preverbal or just-starting-to-speak children. What are the fundamental features or rules that define this subset of a ...
5
votes
2answers
115 views
Where could I find a corpus that is purely descriptive in nature and limited in scope?
I'm trying to build a cognitive model of how people learn a event representation from a sentence describing the event, based on St. John & McClelland's Sentence Gestalt Model.
However, while ...
