Natural Language Processing: Computer programming techniques designed to aid in processing human language.

learn more… | top users | synonyms

2
votes
0answers
10 views

Maltparser vs Freeling for Spanish dependency parser

I need to parse a Spanish corpus with syntactic dependencies. I have tried both Freeling and Maltparser, they both give similar results, but with some divergence. I need the highest accuracy possible, ...
2
votes
0answers
56 views

Shannon's entropy as a measure of vocabulary richness

The Entropy formula for lexical richness is The probability p-ith is calculated by dividing V-ith by N, where N is the total number of tokens in the text and V is the number of types. However, I ...
0
votes
1answer
17 views

Incomplete sentence parsing and relation extraction

I'm building a tool which parses natural language input into nouns and relations. For example sentence Barack is the president of the United States and lives in Washington DC and went to Harvard ...
1
vote
2answers
57 views

How can I represent “I sometimes pet bunnies” using lambda functions?

Update: Removed ancillary wording. My question was how to represent sometimes in propositional logic in the sentence: I sometimes pet bunnies.
1
vote
2answers
43 views

Said tags in Sentence detection

I am trying to apply NLP to fiction but I don't know how split dialog into sentences. Specifically how to split said tags. ex. "Awake? How long have been asleep?" asked teenagers at the same time. ...
1
vote
0answers
25 views

Performance improvements of using the shift-reduce Stanford Parser

I'm running NLP on large texts, and part of the questions I'm trying to answer have to deal with syntax. I'm using Stanford NLP, including the Stanford Parser. I started out using the probabilistic ...
0
votes
0answers
21 views

Average number of independent clauses in a compound sentence

I'm working on a NLP project, where we analyze large text samples (think a novel), and produce some metrics that help us answer interesting questions about the text. One of these metrics is average ...
1
vote
0answers
23 views

Is there a relationship between rank(token) and max(rank(tokens in document))?

Given a language L, a target word w, a [Zipfian] distribution of word frequencies (Rank(w)) and a [gamma] distribution of sentence lengths, can we come up with a way to determine the approximate count ...
0
votes
0answers
9 views

How to discard trash topics from topic models? [migrated]

I am undertaking text analysis of some twitter data. In the end I want to have a data that is interpretable. And so in the end I would like to reduce the data to relevant unit of analysis. Topic ...
1
vote
1answer
45 views

Measuring semantic complexity of a text

I do not have a linguistic background, but I am working on a research project that involves measuring the complexity of news. One of the metrics that I was interested in is the semantic complexity of ...
2
votes
2answers
53 views

What are the most authoratative references on the notion of shallow parsing?

What are the most authoratative references on the notion of shallow parsing (also known as text chunking) in NLP? I am looking for definitions, examples, and discussion of strings of words that ...
1
vote
1answer
52 views

Extract location from a string with a city, state, country in it?

How could I extract a location from a string like San Francisco 49ers and get San Francisco from it? The only way I can is by training a neural-network, but training with every city, state, country, ...
0
votes
0answers
29 views

Early Modern English spelling variations dictionary

I am building a web application that allows users to transcribe, word by word, Early Modern English manuscripts. I want to tell users when a word they transcribed may be a new spelling variant of ...
1
vote
1answer
77 views

What is the name of this text analysis method?

What is the name of a text analysis method where one builds a network of sentences that are connected when they contain words that occur on a predefined reference list of words? So, lets say I have ...
0
votes
0answers
29 views

How to identify mentions in a text?

I am looking for rule-based methods or any other methods to identify all mentions in a text. I have found several libraries that give coreferences but no exact options for only mentions. What I want ...
4
votes
2answers
73 views

Extracting Rare Words From A Corpus

This question might be too general, or opinion-based, but I was looking for advice on how to extract rare words from a very large corpus. The rare words wouldn't necessarily be consistent from ...
-1
votes
1answer
60 views

representing a sentence in integers ( hopefully retaining semantic value )

I've been trying to convert natural language strings into integers for use in a long short-term neural-network. I tried converting to binary, using a bag-of-words, and an associative-array with each ...
1
vote
0answers
36 views

Downloadable list of Romance-language cognates?

I'm looking for tables of cognates between French, Spanish and Italian (and potentially Latin, Portuguese, Catalan, etc.), so I can run some Python on them. If they could have extra information such ...
2
votes
1answer
79 views

Word Co-occurrence and collocation difference?

This Wikipedia page says, "In contrast to collocation, co-occurrence assumes interdependency of the two terms". But I'm not really sure what that means by interdependency of the two terms. Can someone ...
0
votes
1answer
59 views

What should I do of inter-annotator agreement is below 100%?

I am currently working on an NLP project where I had to mark an initial list of words, giving a score of either 1 or 2 to each word in the list. I got the list marked by 2 people and found that the ...
3
votes
0answers
52 views

Inside-outside algorithm (PCFG learning) and binarization

I am implementing the Inside-outside algorithm to estimate the parameters of a PCFG based on the train corpus. One observation (on existing implementations) is that, the grammar is converted into ...
3
votes
2answers
90 views

How to identify corresponding sentences in source and translation texts?

I'm interested in creating files where a source text and translation are interleaved with each other, so that each sentence (or small group of sentences) is immediately followed by its translation. ...
0
votes
1answer
36 views

Semi-offset Annotated Data

I want to use the data provided by CoNLL 2012 shared task. They mentioned that the data is semi-offset annotated, I couldn't find any interpretation for the term "semi-offset annotation". Can anyone ...
0
votes
0answers
68 views

Categorizing short sentences into a taxonomy

I feel like I might be in the wrong exchange, but I do not see a NLP exchange, so I figured this was the best bet. My current task is to take short sentences and try to categorize them based on some ...
1
vote
1answer
93 views

Text Summarization Tool

I'm looking for tools I can use to perform text summarization. Specifically something I could use to summarize personal communication (i.e. email). Preferably I'm looking for something open source ...
3
votes
0answers
73 views

What's the real need for an end-symbol in n-gram models?

There's a footnote in Jurafsky & Martin (2008, p.89) pointing out that, without an end-symbol, an n-gram model would not be "a true probability distribution". Even after seeking the paper they've ...
0
votes
0answers
22 views

NLP: How to do feature normalisation for gender classificiation?

NOTE Before I begin, this F-measure is not related to precision and recall, and its title and definition is taken from this paper. I have a feature known as the F-measure, which is used to measure ...
0
votes
0answers
48 views

How do PCFG parsers actually work?

I'm new to NLP. I have a few doubts about PCFG parser (NLTK). My understanding is that PCFG parser will return most probable parse tree. So if I'm parsing one sentence with PCFG parser, I'll be ...
1
vote
1answer
97 views

Gold Part-Of-Speech tags

I've read this term in many papers in NLP: (Gold POS tags). what does it mean? Thanks.
0
votes
1answer
82 views

What is the most comprehensive context-free grammar for English?

I realize how hard it is to develop a CFG to model an arbitrary English sentence. Some valid discussion points for this topic can be found here: Is there an EBNF that covers all of English? However, ...
0
votes
1answer
117 views

Steps to build a dependency parser [closed]

I'm a beginner in NLP. What are the main steps to build a dependency parser? thanks.
0
votes
0answers
52 views

markov chains in text generation

I'm a beginner in natural language processing, what is the disadvantages of using markov chains in text generation versus using productive grammars? any help or keywords? thanks.
1
vote
2answers
190 views

Detecting actions within text

I'm looking for a way to computationally find any actionable clauses within a body of text. Ideally there is a library I could use that I just give a body of text and it spits out the actions that it ...
0
votes
0answers
95 views

Finding word frequency for calculating PMI

I am going to calculate PMI (Pointwise Mutual Information) for a set of words. As we know, for calculating PMI for two words we need to have the frequency of words (specifically, 1-gram and 2-grams). ...
2
votes
0answers
66 views

What are the most promising research directions for natural language generation?

I have just discovered Natural Language Generation, which seems like a very interesting field. The idea of taking something like highly-structured database records and transforming them into natural ...
2
votes
2answers
95 views

Determine adjective order algorithmically

Does NLTK (or any other Python natural language module) have the ability to determine the order of multiple modifier adjectives? For example: metal round huge bowl (incorrect) huge round metal bowl ...
1
vote
1answer
47 views

What is the linguistic significance of dep and gov methods in Stanford NLP's Typed Dependency class?

I am looking at a TypedDependency object like this: root(ROOT-0, fired-4), which is part of the StanfordNLP suite. If I call the object's .dep() method I get fired/VBD If I call the object's .gov() ...
0
votes
2answers
86 views

Book suggestions for linguistics for Computer Science people

I am asked to do some NLP tasks on a language which is agglutinative. I am finding these terms difficult to understand since my background is different. I am looking for some nice books that give a ...
3
votes
1answer
131 views

Earley Parser: Ambiguity

I've got a pretty basic question concerning the Earley parser: In case of syntactic ambiguity ( S -> NP VP(V NP(NP PP)) vs. S -> NP VP(VP((V NP) PP) ), are both parses stored in one chart or in two? ...
3
votes
1answer
182 views

Combine dictionary lemmatization with stemmer

I'm doing lemmatization as part of a classifier right now. My question is, if it makes sense to first use a dictionary lookup, to find the lemmas (wordnet) and after that additionally, to apply a ...
1
vote
0answers
56 views

VerbNet semantic roles and preposition groups - how to determine matches

Using verbnet to test whether a sentence matches a frame, how does one determine whether the semantic role specified in the verbnet frame is appearing in the sentence or not? e.g. on this verbnet ...
0
votes
0answers
14 views

Identifying styles of reasoning in text

I'm trying to programmatically identify in a text some elements of "style of reasoning". I'm more specifically thinking about syllogism, induction, abduction, deduction, etc. More generally i would ...
1
vote
1answer
73 views

What is state of the art in the field of NLP? [closed]

I am new to Natural Language Processing, I think NLP is a challenging field, the syntax and semantic ambiguities could cause a lot of problems. For example I think for these problems machine ...
1
vote
0answers
29 views

Any NLP performance timeline?

I am looking for any plots, etc. that show performance metrics on various NLP tasks as a function of year. E.g. [the following ...
2
votes
1answer
502 views

What do WordNet::Similarity scores mean?

I am using WordNet Interface in NLTK, which facilitates computation of a number of similarity metrics: Path similarity Leacock-Chodorow Similarity Wu-Palmer Similarity Resnik Similarity ...
0
votes
0answers
53 views

NLP system with end-to-end ambiguity support?

I've noticed that most NLP systems I can easily investigate on the Internet don't handle ambiguity at all, or only handle within components of the pipeline and not between components. For instance it ...
1
vote
1answer
222 views

NLP conversion between parts of speech and pertainyms?

I would like to write a program that can automatically group e.g. 'happiness', 'happily', 'happy' into 'happy': What do I need to read to get a handle on this subject? What is it called? What is the ...
0
votes
0answers
46 views

adjective features - similar to beth Levin's verb classes?

Beth Levin published research on verb classes/alternations - is there similar for the adjective category of words? I can see references to countable/uncountable, gradable,absolute, of quantity/quality ...
0
votes
0answers
41 views

Where to find a dataset of related/sister terms

I'm working on a Natural Language Processing project that requires me to automatically categorize various articles. I've tried creating categories based on wikipedia, freebase and dbpedia but often ...
1
vote
5answers
304 views

Introduction to linguistics from a practical application standpoint

I currently have a project that I need to do that involves quite a bit of computational linguistics. My background in linguistics is very high level - I wonder if there are any good introductory ...