6

I am not an expert in linguistics at all; more of a physicist instead. So I don't know if there are any defined distances on words D(W1, W2) that really represents how the human memory works; for example 2 nearby words are to be more likely to be misremembered instead of each other or something like that. Do you guys know any distance that is so?

I found the Levenshtein distance on the internet but I don't know if it does what I want (a study that shows it does is also a good answer to my question).

Improve this question
3
  • 1
    Some quick thoughts: Do you need your distance measure to be symmetric? (In practice, W1 -> W2 may not be the same as W2 -> W1) Are you talking about isolated words, or words in use? (Discourse context, syntax context, and meaning change the distances) For whom? (The distance would probably differ a lot by individuals, at least by vocabulary.) Relevant to Levenshtein distance (and similar measures), are you considering ’surface' forms or lemmas (e.g. cat, cats have the lemma ‘cat’)?
    – Jeremy Needle
    Apr 19, 2020 at 13:33
  • Well I didn't understand all of the things you just said, but I don't care about symmetry and I am talking about out of context words.I know individuals are different, but perhaps some rough general distance may be defined. Thank you for the quick response.
    – K. Sadri
    Apr 19, 2020 at 14:05
  • I don't understand what you mean by "instead of each other" - can you clarify?
    – amI
    Apr 20, 2020 at 20:46

3 Answers 3

Reset to default
3

What you remember is quite vague, but I think this is related to word vectors. Word vectors are internal representations of words learned by a neural network, and they live in a high dimensional Euclidean space (typically several hundred dimensions).

One algorithm to get at word vectors is word2vec. It is still poorly understood why the word vectors have the features that we can observe, mainly because of the inherent opaqueness of the neural network.

Improve this answer
6
  • 1
    It's true that this is the answer to the question, but it's untrue that the mechanism of action is poorly understood; the architecture of the various techniques to achieve this result were not discovered, but engineered.
    – TheLoneDeranger
    Apr 20, 2020 at 23:35
  • We understand the design of a neural network but what happens during the training phase of the NN is the opaque part.
    – Sir Cornflakes
    Apr 21, 2020 at 9:38
  • 1
    What happens during the training phase is precisely what the networks were designed to do during the training phase; there's nothing unknown about the training of cbow, skipgram, word2vec, and other such methods. It's true that what any particular neuron value will be, at any given iteration in the training phase, unknown, but there is no intermediate data involved in these methods; the action managing the changes of neuron values over training is quite discernible, and, again, was the basis of these networks' inventions.
    – TheLoneDeranger
    Apr 21, 2020 at 21:54
  • So you are ready to explain to mere mortal why an equation like |king⟩+|woman⟩-|man⟩≃|queen⟩ holds for word2vec word verctors, because you understand the process fully?
    – Sir Cornflakes
    Apr 21, 2020 at 22:17
  • Yes, absolutely, although it's too much for a comment. It occurred to me that the confusion of our conversation may be such as this: the features of intermediate layers in deep neural networks is typically largely indecipherable, and the particular actions of randomly initialized neurons at any given point in training aren't even worth bothering to try deciphering. However, since there is no intermediate data in these manifold learning techniques, the neuron features can be described as precisely that of the action of the network.
    – TheLoneDeranger
    Apr 21, 2020 at 22:18
1

One can use the cosine similarity measure in word space. Google up some papers by Mikolov, who describes how the models can be generated. It’s quite remarkable that they can derive, for example, |king⟩+|woman⟩-|man⟩≃|queen⟩.

Improve this answer
1
  • That sure sounds interesting. I'm going to look it up and wait a few more days, then perhaps I will close the question with your answer.
    – K. Sadri
    Apr 20, 2020 at 10:28
0

Levenstein distance is computed as the number of elements that need to be exchanged to switch from a starting sequence to another (D_L(0010, 3000)=2). This is permutation and each exchange is atomic, so it does not even respect how close or far appart the features of the changing elements are, or whether the elements are atomic parts of speach (which they are not).

Similarity of words is investigated empiricly, for starters. I only know of one series of experiments on lexicalization in first language learning, who has found that, say peg would not be confused easily for dog depending on the context (and I'd love to name a reference work; I cannot give a qualified summary either way, especially regarding what context); whereas, I suppose, in case of impoverished context that would be most sever in aphasia, the inadvertant confusion of phonemes might be possible, I'm sure.

Yet, all that says nothing about confusion that depends on context. Lexical distance is quite something else, not precisely defined, though Word2Vec, as JK indicates above, is one famous approach leaning on distributional semantics, though slightly mechanic.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.