Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
29
votes
5answers
1k views

Lists of linguistic resources

In the interest of cultivating a professional, academic community, I posted this question on Meta. One comment was to open a community-wiki question inviting others to contribute to a list of academic ...
25
votes
2answers
45k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
20
votes
4answers
2k views

Corpus of baby-talk or motherese

Baby-talk or motherese is the language that parents tend to use when addressing preverbal or just-starting-to-speak children. What are the fundamental features or rules that define this subset of a ...
18
votes
1answer
8k views

Does anyone know of text message corpora?

I am looking for a large corpus of text messages. By large, I am hoping to have at least 15,000 text messages in my sample. I am fine with combining several smaller corpora into a larger corpus as I ...
14
votes
4answers
5k views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
13
votes
3answers
524 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
12
votes
1answer
617 views

What are some resources that I can use to gather Twitter data for an NLP project?

I was going to use the Edinburgh Twitter Corpus that is referenced in this paper: The Edinburgh Twitter Corpus. But apparently Twitter has changed their Terms and the corpus is no longer available. ...
11
votes
2answers
589 views

How to best clean a large historical corpus ridden with OCR errors

Overview: I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some ...
11
votes
3answers
3k views

Where to find frequency list of English words from newspapers, books and magazines?

I'd like to download a frequency list of English words that are most common in newspapers, books and magazines. It should contain at least 50.000 words. Is it possible to download such a list from ...
10
votes
1answer
566 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
9
votes
3answers
1k views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...
9
votes
1answer
399 views

What is the Relationship Between Document Length and Unique Words

Sorry if my question is a little more mathematical in nature, but my question is: Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
9
votes
1answer
10k views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
9
votes
1answer
229 views

Is there an automatic way of identifying transitive verbs in Computational Linguistics?

Is there any straightforward way of identifying transitive verbs (or sentences containing transitive constructions) in an BrE English text? I've looked into semantic shallow parsers, such as Semafor, ...
9
votes
3answers
2k views

How to work on annotating AND sentence-aligning parallel texts?

There are plenty of software programs facilitating interlinear transcription (e.g. Toolbox, Fieldworks Language Explorer). There's also a number of tools that allow you to work on aligning parallel ...
8
votes
7answers
3k views

Biggest freely available English corpus?

Any help on finding the biggest freely available English corpus that can be used on research? So far I have found OANC with 15 M words.
7
votes
1answer
186 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
7
votes
2answers
944 views

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Alternatively, do any larger corpora have categories that pertain to technology, computer science, etc.? Or collections of English documentation and/or manuals related to modern devices, interfaces, ...
7
votes
1answer
226 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...
7
votes
1answer
728 views

How can I calculate if the difference between two word frequencies in one corpus is significant?

I want to study orthographical variants, for example: Can firefighter and fire-fighter be considered orthographical variants (i.e. the difference in frequency in a corpus is not statistically ...
7
votes
2answers
964 views

Do I have copyright issues when making a corpus from the web?

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I ...
7
votes
3answers
958 views

Are there dictionaries like Collins COBUILD for other languages than English? [closed]

Let me explain what I'm after: monolingual all examples are taken from a large corpus of contemporary text (the COBUILD series is now using Bank of English) the senses for each entry are sorted by ...
6
votes
2answers
364 views

Where could I find a corpus that is purely descriptive in nature and limited in scope?

I'm trying to build a cognitive model of how people learn a event representation from a sentence describing the event, based on St. John & McClelland's Sentence Gestalt Model. However, while ...
6
votes
3answers
2k views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
6
votes
1answer
54 views

Corpus of English sentences written by non-native speakers?

I am working on a hobby project which tends to help the non-native speakers figure out if the sentences they write are good enough and sounds like written by English native speakers. Any relevant info ...
6
votes
2answers
3k views

How to establish word frequencies from a corpus?

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
6
votes
1answer
445 views

How can I know if a log-likelihood score is high enough?

I am trying to understand if some terms can be considered as distinctly American or British. So for example I was studying the term "aubergine" and the normalized frequency in the American corpus is ...
6
votes
1answer
112 views

How to make a Corpus? (selecting a good structure)

I'm making a Corpus. I have searched for good practices on how to build it. I've downloaded some Corpus to check their internal structure and the extension file, but I didn't find many. So, I am lost ...
6
votes
0answers
71 views

Is there a threshold of usage of a word, below which it will become an archaism?

University student, beginning to learn about corpora, developing a research question for an assignment. There may well be linguistic concepts and terminology that already cover this idea and that I am ...
5
votes
1answer
114 views

What are the conventions for corpus usage?

Suppose I'm working with a language that doesn't have much data available; the largest corpus around is a translation of the Bible, which is free to read online, but is presumably still copyrighted by ...
5
votes
1answer
126 views

Parts-of-speech tagging and finding relevant phrases in documents

I've got a corpus of half a million text documents. I'd like to identify phrases in each document that are the most descriptive with which to build tag clouds. Let's say that I identify the most ...
5
votes
1answer
109 views

Word length as a function of word frequency?

I was wondering if any studies have been done in corpus linguistics which look at word length (number of phonemes per word) as a function of the word's frequency of use? Thanks! EDIT: Never mind, ...
5
votes
1answer
101 views

How should grammatical errors be treated in parsing and tokenisation?

I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are ...
5
votes
1answer
409 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
5
votes
2answers
177 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
5
votes
3answers
604 views

Are there tools to build domain specific corpus? [closed]

I would like to build a domain specific corpus. By this I mean a corpus from a specific knowledge area, for instance Law, Psychiatry, or Klingon language. I am currently using a Wikipedia corpus, and ...
5
votes
2answers
345 views

Corpus Linguistics: Is it possible to add a tag for “sentence ending”?

I'm new to Corpus Linguistics and I'm writing a paper about the English and Portuguese "because noun", a type of construction such as "I'm going home because GTA5". However, when I try to search this ...
5
votes
1answer
340 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
5
votes
1answer
807 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
5
votes
2answers
786 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
4
votes
1answer
422 views

Is there a standard corpus against which to benchmark mechanical parsers?

I was wondering if there is a published standard corpus of English sentences which are confounding for mechanical parsers. I am looking over sentences given here, and I have the following (very short) ...
4
votes
2answers
336 views

JSON for corpora

Do you know any cases of corpus linguistics tools or corpora, using JSON to "mark" or delimited linguistics units in morphology, syntax,semantics? It seems all marking systems use some kind of XML or ...
4
votes
1answer
48 views

Looking for Spanish email or chat corpora

I am looking for some corpora containing emails or chats in Spanish.
4
votes
2answers
173 views

When analyzing a set of corpora, are there any standard practices with regard to the classification of gerunds?

In the article, "How Many Words Do You Need to Know in Spanish (or any other foreign language)? And WHICH Words Should You Be Learning?" I came upon the following: “Assume that a language learner ...
4
votes
3answers
2k views

Looking for korean text corpus

I'm looking for a list of text corpus for korean. Is there any web site where I can find it? Regards.
4
votes
1answer
204 views

A classical text about the Sun?

I've read once there is a text about the Sun, which is created to use words and concepts which shall be present in any human language, and which is translated nearly to all human languages for using ...
4
votes
1answer
2k views

Is there a computational method to syllabify English words?

There are straightforward ways to convert English words to phonemes via a dictionary that contains such information. However, is there a way to automatically convert English text into syllables? I.e., ...
4
votes
3answers
1k views

Word frequency list for agglutinative languages like Swahili?

I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
4
votes
3answers
2k views

Where can I find a free text corpus for the Hindi language?

I need a text corpus for my NLP project in Hindi, however all the corpora I found are not free for downloading. Is there any free annotated Hindi text corpus at all? If yes, where can I find it?
4
votes
3answers
207 views

American English Corpus with spoken language exchange

I'm looking for an American English Corpus with spoken language exchange. There are many terms used. E.g. eTandem, tandem, etc. TIA

1
2 3 4 5