Hot answers tagged

19

The most accessible resource that explains the difference between each of these word similarity metrics would be Dan Jurafsky and James H. Martin's ubiquitous Speech and Language Processing 2nd Edition. Specifically, pages 652-667 in chapter 20 (Computational Lexical Semantics) briefly and comprehensively cover each metric/algorithm in a way that anyone ...


9

I guess the NLTK documentation is a bit off. Looking at Wordnet's documents, I see: pos Syntactic category: n for noun files, v for verb files, a for adjective files, r for adverb files. And in another section of the same document: ss_type One character code indicating the synset type: n NOUN v VERB a ADJECTIVE ...


4

WordNet is not a dictionary but a semantic lexical database. The key function of WordNet is to create a network of semantic relationships between words (synonyms, meronyms, etc.) So it makes sense it would only focus on content words and not function words (which is what stop words are). There are many free online dictionaries - depends on what you need one ...


3

I think the result you can find out there would be the experimentation of someone else. So my suggestion is make your own experimentation: build your own classifier. A good and simple idea collect as many classified data as you can choose the similarity as a feature and peek the labels as similar and non-similar And, assuming the data is linearly ...


3

I have checked on Wordnet, and gasoline is indeed listed as a substance, while wine and milk are listed as food. Wordnet and other such ontologies impose taxonomies on the world, and the thing about taxonomies is, you can classify entities more than one way. If an artefact is understood as something that only exists in the world through human intervention, ...


3

No. Treebanks are about sentences analysed with syntax trees (constituent and/or dependency parsing). Wordnets are about semantic relation between words like hypernymy and hyponymy. The two resource types have different objects as their contents and they are analysing different kinds of relations between those objects. Besides being linguistic resources ...


2

Given your tags, I assume you're using the nltk library? First you'll want the Reuters and Wordnet corpora: from nltk.corpus import reuters, wordnet Then, get your words from the Reuters corpus however you like: you probably don't want the entirety of reuters.words(), so you should narrow it down by field, file, dataset, etc. Then, find how many senses ...


1

Here's one way you could do this: Find a suitable corpus; there are plenty out there, but which one is best depends on your specific application. This'll be easier for English than for most other languages, and the larger the corpus, the better the results will be. Use a word2vec model on the corpus. This is a simple neural network that transforms words ...


1

AFAIK none, the original Wordnet was built up by hand from scratch.


1

Here is the reference: In WordNet only the base forms of words are generally represented. ... [With regards to phrases] In general, only the base forms of words, even those making up compounds such as attorney general, are stored in WordNet from: The chapter: Design and implementation of the WordNet lexical database and searching software, by ...


1

I don't know of a toolset, but I can think of a factors that you'd need to account for to determine 'relatedness' across parts of speech: First, consider for the eight parts of speech: verb adjective adverb noun pronoun preposition conjunction interjection I'm taking a guess at this, but I'd say that not all eight parts of speech communicate an equal ...


1

I suppose it depends on the task. If you don't often expect to see out-of-vocabulary words (i.e., words for which you don't have lemmata), there is no point in stemming—in fact, it might be harmful in that it might normalize two words of different meaning to the same stemmed string. Yes, stemming can be harmful. It'd be best to try lemmatization vs. ...


1

I assume you're looking for an electronic database (as opposed to the rich tradition of character analysis in paper dictionaries and such). KanjiVG is based on character appearance and includes data on the components of each kanji; it's easy to parse the XML to do things like finding all kanji with a given component, etc. Sorry for the self-plug, but I ...


1

Try using the English Wiktionary. You can download a dump from here; look for enwiktionary. Its structure is quite straight-forward, so it should not be too much of a problem to automatically remove all of the non-English entries and re-format it in a structure you can use for your purpose. But, …, may I ask why do you want to create a multilingual online ...


1

See this WordNet entry for bus. The senses are already ranked according to how common they are. In addition to that, the first (optional) field, between parentheses, gives an indication of how frequently each word-sense was seen. Frequency of use is determined by the number of times a sense is tagged in the various semantic concordance texts. I don't know ...


Only top voted, non community-wiki answers of a minimum length are eligible