Questions tagged [corpora]

A corpus (plural corpora) is a database of text or speech, possibly annotated with language-specific explanatory information. Used for testing statistical hypothesis and constructing quantitative models.

Filter by
Sorted by
Tagged with
29 votes
2 answers
49k views

English text corpus for download

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I ...
user avatar
  • 1,022
28 votes
5 answers
1k views

Lists of linguistic resources

In the interest of cultivating a professional, academic community, I posted this question on Meta. One comment was to open a community-wiki question inviting others to contribute to a list of academic ...
20 votes
4 answers
2k views

Corpus of baby-talk or motherese

Baby-talk or motherese is the language that parents tend to use when addressing preverbal or just-starting-to-speak children. What are the fundamental features or rules that define this subset of a ...
user avatar
19 votes
3 answers
9k views

Does anyone know of text message corpora?

I am looking for a large corpus of text messages. By large, I am hoping to have at least 15,000 text messages in my sample. I am fine with combining several smaller corpora into a larger corpus as I ...
user avatar
  • 417
14 votes
4 answers
6k views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
user avatar
  • 522
12 votes
1 answer
636 views

What are some resources that I can use to gather Twitter data for an NLP project?

I was going to use the Edinburgh Twitter Corpus that is referenced in this paper: The Edinburgh Twitter Corpus. But apparently Twitter has changed their Terms and the corpus is no longer available. ...
user avatar
12 votes
3 answers
559 views

Best method for building a learner corpus for DDL

I'm looking for a set of free and (somewhat) easy tools that I can use with my EFL writing students next semester. I want to analyze their initial essays and look for common errors that can be ...
user avatar
11 votes
2 answers
612 views

How to best clean a large historical corpus ridden with OCR errors

Overview: I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some ...
user avatar
  • 211
11 votes
3 answers
4k views

Where to find frequency list of English words from newspapers, books and magazines?

I'd like to download a frequency list of English words that are most common in newspapers, books and magazines. It should contain at least 50.000 words. Is it possible to download such a list from ...
user avatar
  • 261
10 votes
1 answer
631 views

The power of trigram language models (2nd order Markov models)

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on ...
user avatar
  • 367
9 votes
7 answers
5k views

Biggest freely available English corpus?

Any help on finding the biggest freely available English corpus that can be used on research? So far I have found OANC with 15 M words.
user avatar
  • 101
9 votes
3 answers
1k views

Tools to annotate (categorise) sentences from a sentence corpus

I have a corpus consisting of sentences that are to be categorised in order to train a text categorisation algorithm. I am looking for a (preferably web-based) tool that: Allows me to input a list ...
user avatar
  • 231
9 votes
1 answer
471 views

What is the Relationship Between Document Length and Unique Words

Sorry if my question is a little more mathematical in nature, but my question is: Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of ...
user avatar
9 votes
1 answer
10k views

Corpus of Chat/IM/Text Conversations?

I was wondering if there is any conversational corpus available to the public. The ideal corpus would be one made up of AIM messages with users tagged and lots of different users. I would imagine ...
user avatar
  • 191
9 votes
1 answer
241 views

Is there an automatic way of identifying transitive verbs in Computational Linguistics?

Is there any straightforward way of identifying transitive verbs (or sentences containing transitive constructions) in an BrE English text? I've looked into semantic shallow parsers, such as Semafor, ...
user avatar
  • 367
9 votes
3 answers
2k views

How to work on annotating AND sentence-aligning parallel texts?

There are plenty of software programs facilitating interlinear transcription (e.g. Toolbox, Fieldworks Language Explorer). There's also a number of tools that allow you to work on aligning parallel ...
user avatar
8 votes
1 answer
201 views

How are dictionaries produced

What are the steps invoked in producing a dictionary? I am primarily interested in understanding the role software plays in the production process. Obviously a corpus for the language is first ...
user avatar
  • 1,022
7 votes
1 answer
235 views

Is there a digital corpus somewhere of pre-Latin Vietnamese text?

I like as a hobby to do text processing and analysis of CJKV text. Japanese and Chinese is of course available in near infinite quantities. Korean is a bit harder to find, but not impossible. ...
user avatar
  • 14.3k
7 votes
1 answer
845 views

How can I calculate if the difference between two word frequencies in one corpus is significant?

I want to study orthographical variants, for example: Can firefighter and fire-fighter be considered orthographical variants (i.e. the difference in frequency in a corpus is not statistically ...
user avatar
7 votes
3 answers
976 views

Are there dictionaries like Collins COBUILD for other languages than English? [closed]

Let me explain what I'm after: monolingual all examples are taken from a large corpus of contemporary text (the COBUILD series is now using Bank of English) the senses for each entry are sorted by ...
6 votes
2 answers
377 views

Where could I find a corpus that is purely descriptive in nature and limited in scope?

I'm trying to build a cognitive model of how people learn a event representation from a sentence describing the event, based on St. John & McClelland's Sentence Gestalt Model. However, while ...
user avatar
  • 263
6 votes
1 answer
126 views

What are the conventions for corpus usage?

Suppose I'm working with a language that doesn't have much data available; the largest corpus around is a translation of the Bible, which is free to read online, but is presumably still copyrighted by ...
user avatar
  • 53.2k
6 votes
2 answers
1k views

Is there any corpus for technical English? (E.g., computers, IT, modern technology)

Alternatively, do any larger corpora have categories that pertain to technology, computer science, etc.? Or collections of English documentation and/or manuals related to modern devices, interfaces, ...
user avatar
6 votes
3 answers
2k views

Tool for building a corpus by crawling the web?

Often I want to make my own corpus of a new language I've become interested in. I'm very happy to make my own tools and have plenty of programming experience. I have made my own tool to gather plain ...
user avatar
  • 14.3k
6 votes
1 answer
68 views

Corpus of English sentences written by non-native speakers?

I am working on a hobby project which tends to help the non-native speakers figure out if the sentences they write are good enough and sounds like written by English native speakers. Any relevant info ...
user avatar
6 votes
1 answer
146 views

Word length as a function of word frequency?

I was wondering if any studies have been done in corpus linguistics which look at word length (number of phonemes per word) as a function of the word's frequency of use? Thanks! EDIT: Never mind, ...
user avatar
  • 724
6 votes
1 answer
637 views

How can I know if a log-likelihood score is high enough?

I am trying to understand if some terms can be considered as distinctly American or British. So for example I was studying the term "aubergine" and the normalized frequency in the American corpus is ...
user avatar
  • 63
6 votes
1 answer
236 views

How to make a Corpus? (selecting a good structure)

I'm making a Corpus. I have searched for good practices on how to build it. I've downloaded some Corpus to check their internal structure and the extension file, but I didn't find many. So, I am lost ...
user avatar
  • 63
6 votes
1 answer
4k views

How to establish word frequencies from a corpus?

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
user avatar
  • 1,022
6 votes
2 answers
210 views

Corpora of Indigenous American Languages?

Does anyone know of any (preferably free) online corpora of indigenous American languages (anywhere on the American hemisphere)? I've managed to find corpora of many of the common world languages, but ...
user avatar
6 votes
2 answers
1k views

Do I have copyright issues when making a corpus from the web?

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I ...
user avatar
  • 173
6 votes
0 answers
76 views

Is there a threshold of usage of a word, below which it will become an archaism?

University student, beginning to learn about corpora, developing a research question for an assignment. There may well be linguistic concepts and terminology that already cover this idea and that I am ...
user avatar
  • 181
5 votes
3 answers
953 views

How can I find the frequency for the Russian word "чем-нибудь"?

I've tried searching for the word "чем-нибудь"on the 20,000 word list I found on Wikipedia. You can link directly to it here. For the sake of thoroughness, I even searched the larger 50,...
user avatar
  • 313
5 votes
1 answer
129 views

Parts-of-speech tagging and finding relevant phrases in documents

I've got a corpus of half a million text documents. I'd like to identify phrases in each document that are the most descriptive with which to build tag clouds. Let's say that I identify the most ...
user avatar
  • 181
5 votes
3 answers
3k views

Looking for korean text corpus

I'm looking for a list of text corpus for korean. Is there any web site where I can find it? Regards.
user avatar
5 votes
3 answers
2k views

Where can I find a free text corpus for the Hindi language?

I need a text corpus for my NLP project in Hindi, however all the corpora I found are not free for downloading. Is there any free annotated Hindi text corpus at all? If yes, where can I find it?
user avatar
  • 173
5 votes
1 answer
511 views

Where online can I find a list of all the Hapax Legomena in the Hebrew Bible?

I am looking for a list online of all Hapax Legomena in the Hebrew Bible in text format that I can copy and paste into MS Word for further studies. I need something that's free and is not in a picture ...
user avatar
5 votes
1 answer
107 views

How should grammatical errors be treated in parsing and tokenisation?

I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are ...
user avatar
5 votes
1 answer
417 views

Semantic frame representation of ATIS 3 corpora

Air Travel Information Service (ATIS) is a benchmark corpora/dataset in spoken language understanding field. Many papers make use of it to see how their algorithms perform. There are many tagging ...
user avatar
  • 283
5 votes
3 answers
657 views

Are there tools to build domain specific corpus? [closed]

I would like to build a domain specific corpus. By this I mean a corpus from a specific knowledge area, for instance Law, Psychiatry, or Klingon language. I am currently using a Wikipedia corpus, and ...
user avatar
  • 171
5 votes
2 answers
391 views

Corpus Linguistics: Is it possible to add a tag for "sentence ending"?

I'm new to Corpus Linguistics and I'm writing a paper about the English and Portuguese "because noun", a type of construction such as "I'm going home because GTA5". However, when I try to search this ...
user avatar
  • 335
5 votes
1 answer
366 views

Ranking sentences

I have a list of 15000 sentences for a new language I wish to learn. I also have the English translation of each of these sentences. Additionally, I have a 30 million word corpus for the new language. ...
user avatar
  • 51
5 votes
1 answer
839 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
user avatar
  • 1,022
5 votes
1 answer
203 views

Where can I find training data for dialects of Hindi?

I am working on a NLP project that aims at identifying different dialects of Hindi language. My über goal is to generate dialogs in once the program identifies which dialect is given. I tried ...
user avatar
5 votes
2 answers
859 views

Is there a corpus of English adjective, adverb, etc metadata?

For example. In the sentence fragment "red leather stool", red is a color, leather is a covering of the stool, and stool is a chair. Is there any database, corpus or repository, that lists the ...
user avatar
4 votes
3 answers
3k views

Conversational English corpus for download

I have a project which requires a corpus of conversational English in plain text (although I can perform some processing as needed). Since I am a student, I need to find a corpus that is free and ...
user avatar
  • 159
4 votes
1 answer
453 views

Is there a standard corpus against which to benchmark mechanical parsers?

I was wondering if there is a published standard corpus of English sentences which are confounding for mechanical parsers. I am looking over sentences given here, and I have the following (very short) ...
user avatar
4 votes
2 answers
508 views

JSON for corpora

Do you know any cases of corpus linguistics tools or corpora, using JSON to "mark" or delimited linguistics units in morphology, syntax,semantics? It seems all marking systems use some kind of XML or ...
user avatar
4 votes
1 answer
66 views

Looking for Spanish email or chat corpora

I am looking for some corpora containing emails or chats in Spanish.
user avatar
4 votes
2 answers
183 views

When analyzing a set of corpora, are there any standard practices with regard to the classification of gerunds?

In the article, "How Many Words Do You Need to Know in Spanish (or any other foreign language)? And WHICH Words Should You Be Learning?" I came upon the following: “Assume that a language learner ...
user avatar
  • 313

1
2 3 4 5