Hot answers tagged

19

The most accessible resource that explains the difference between each of these word similarity metrics would be Dan Jurafsky and James H. Martin's ubiquitous Speech and Language Processing 2nd Edition. Specifically, pages 652-667 in chapter 20 (Computational Lexical Semantics) briefly and comprehensively cover each metric/algorithm in a way that anyone ...


4

WordNet is not a dictionary but a semantic lexical database. The key function of WordNet is to create a network of semantic relationships between words (synonyms, meronyms, etc.) So it makes sense it would only focus on content words and not function words (which is what stop words are). There are many free online dictionaries - depends on what you need one ...


4

Your question is tagged for NLTK, but if you're free to switch, Stanford NLP has a product called Tregex which does exactly this kind of tree search. You might also be able to shell out to the Tregex binary, which has a fairly useful CLI as well. See documentation on TregexPattern#main for more. You can find some introductory documentation here. (...


3

There are quite a few tools which can do this. For example: Tregex mentioned by Jon Gauthier. I know a few others: PML-TQ https://ufal.mff.cuni.cz/pmltq/ Tundra https://weblicht.sfs.uni-tuebingen.de/tundra-beta Iness search Among them, I know Tundra best since I used to work in the group who created it. Tundra has a very nice UI based on bootstrap, as ...


3

There are two paths you can take to solve this problem, both outlined in Jurafsky & Martin (2016), Ch. 15. 1) If computing your own semantic similarity matrix, you can shrink the size of the window where the model looks for co-occurrences. In your example, if the window around the title was 3 words, it would pick up "quite good" but not "disliked." In ...


3

Your broader question is about semantic relations, but the example you gave was about a narrower, but still widely researched field, that of opinion mining. Since you're asking about an entire field, it would be hard for me to get into the details. I'll treat the question as a reference request, and recommend the book Opinion mining and sentiment analysis. ...


3

In the first case, you're training with the entire data set, then testing on different chunks of it. In the second case, you're training with chunks of the data set, then testing on the whole thing. You probably don't want to be doing either. You want to train on as much data as possible, but you never want to test on the same data you trained with (if you ...


3

Chapter 5 of the online NLTK book explains the concepts and procedures you would use to create a tagged corpus. There are several taggers which can use a tagged corpus to build a tagger for a new language. You will probably want to experiment with at least a few of them. A tagged corpus is better than just a list of words because many languages have ...


3

Any context-free parser can be used. However "pure" CF grammars aren't practical for real applications. I'd recommend to use LFG or a similar tool that generates more useful underlying representations.


2

Based on your comment, you seem to be looking for positive words with the intent of performing sentiment analysis (generally lists are divided into positive, negative, and neutral). Here are some lists to get started with (a Google search will turn up plenty more): Hu and Liu, KDD, 2004 (~6800 words) Breen, Twitter Sentiment Analysis Tutorial, 2011 Davies, ...


2

The issue of completing incomplete sentences has actually been adressed formally. There are many way to view it. Modifying the grammar is a possibility, but not the best method in my opinion, at least if you are too crude about it. There are very generic techniques for ill-formed input that can be used to modify in various ways a non-grammatical input ...


2

You are implementing the Jaccard coefficient whereas the library has the Jaccard distance. The coefficient tells how related two sets are (it is high when they are similar), whereas the distance does the opposite; it is low when they are similar. In fact, they are each other's complement, i.e. d = 1-c and c = 1-d. This is also explained on the Wikipedia ...


2

The NLTK book, chapter 10, which provides some theoretical background to the implementation, references NLTK ch. 9, where the feature grammar is introduced. Have you read these articles? Ch. 9 explains all the the features you were asking about including the ? notation. Ch. 10 introduces the lambda calculus. The special function app in the ...


1

For your first two sentences, my intuition is that the attachment of the PP subtly alters the semantics with the result that in some circumstances only one attachment is appropriate. We start with your two sentences: Twain [bought [a book [for Howells]]]. Twain [bought [a book] [for Howells]]. First imagine that Howells has told Twain to buy a book and ...


1

This blogpost on LingPipe gives a fine overview over chunk encoding. It says explicitly over the IO encoding (that is IOB without B) The simplest encoding is the IO encoding, which tags each token as either being in (I_X) a particular type of named entity type X or in no entity (O). This encoding is defective in that it can’t represent two entities next to ...


1

Yes, from the perspective of that manual the second answer is the right one. 2.1 Lists What is a text? At one level, it is a sequence of symbols on a page such as this one. At another level, it is a sequence of chapters, made up of a sequence of sections, where each section is a sequence of paragraphs, and so on. However, for our purposes, we will ...


1

Define “meaningful”. Implement that definition by means of a filter that will accept meaningful text, i.e., text that can be understood under such rules. A modern way to solve this problem would be to train a neural network of some sort with a large set of phrases that are “meaningful” (e.g., famous quotes and excerpts from books) and a set of randomly-...


1

As stated here, the corpus is a bunch of XML files in which the authors are encoded as an attribute value to a post element: To view the individual XML files in an editor (because this will help you understanding their strcture), just go to the directory where it is stored (default directories are given here). If you want to use it in Python: nps_chat....


1

Do you actually need to come up with the properties for your adjectives, or is the goal to determine whether some order is valid (and properties are just a way of making that determination)? If the goal is to simply say that e.g. "metal round huge bowl" is incorrect, then you can skip the tagging of your adjectives for their properties entirely and go ...


1

I don't know of a toolset, but I can think of a factors that you'd need to account for to determine 'relatedness' across parts of speech: First, consider for the eight parts of speech: verb adjective adverb noun pronoun preposition conjunction interjection I'm taking a guess at this, but I'd say that not all eight parts of speech communicate an equal ...


1

This grammar in the article is ambiguous, and the article says that ambiguity is part of the design. Hence you need a parser that can handle ambiguity. Many context-free parsers will not do that. However, NLTK does offer a chart parser, i.e., an algorithm (based on dynamic programming) that can parse any context-free grammar and give you ways of dealing ...


Only top voted, non community-wiki answers of a minimum length are eligible