I want to build a keyword extractor based on the TextRank model as explained in RMPT04. But I don't understand how to calculate the co-occurrence between two words in a window of text explained in the point 3.1. Moreover, is a corpus necessary?

  • 1
    this should be moved to compling or statistics. – jlawler May 5 '13 at 16:07
  • 1
    @jlawler: I disagree. There is no compling. nlp is a rather neglected topic on statistics.SE. Until NLP.SE starts, this question should be considered on-topic here. – prash May 6 '13 at 14:49
  • @ArnoldPaye: The "window" here is just an indication of how many intermediate words (in a sentence) are to be ignored in order to consider two words as "co-ocurring". Given the sentence, "Mary had a little lamb", "Mary" and "lamb" are considered to co-occur of the window is >=4. They are considered as not co-occurring if the window is <4. These co-occurrence counts are extracted over a corpus: a corpus is necessary. – prash May 6 '13 at 15:37
  • @prash Whether NLP SE starts or not, it doesn't influence our site. NLP questions will always be on topic on this site regardless of that site existence. If we had to cut topics that overlapped, most sites would had a really narrow field of expertise in their FAQ. :) – Alenanno Jun 26 '13 at 13:49
  • Note that the link to the referenced article may go stale making this question void. Please add enough context to make the question self-contained. – jknappen Mar 13 '17 at 14:05

Of course you need a corpus.

Generally in statistical NLP, you train your model based on a corpus. For example, for text classification where an input document is fed to the model and it should output its class (from a list of classes). The model is trained on many documents with their corresponding classes and when the new document is tested under that model, it will use the features (information) which was extracted from those documents to classify the new document.

In the case of co-occurrence of two words, you can use context-vector, which is very common in statistical NLP. It has a simple definition and very easy to implement, but you will need a corpus:

You will define a vector with fixed length (the number of unique words in your corpus) for each unique word in your corpus. The context vector for each word tells us how many times other words have co-occurred with the current word in the defined window, e.g. in a window of words, you see what are the other words occurred with the current word and increment their corresponding element in the context vector. A simple example is show below:

Corpus: A D C E A D F E B A C E D

Window size: 2 (the 2 words of the either side)

Context vectors:

      A B C D E

A     0 1 3 2 3    
B     1 0 1 0 1     
C     3 1 0 2 2    
D     2 0 2 0 4    
E     3 1 2 4 0

Using these context vectors you can get co-occurrences very easy. For example co-occurrence of D and E is D[E] = 4.

  • sorry but I dont get how D[E] got 4 and also others like A[E] gets 3? – samsamara Apr 28 '15 at 2:44
  • @KillBill If you consider the window around all the occurrences of A, you will find 3 E's co-occuring with A. The same thing goes with D, if you consider the window around each occurrence of D, and count all the occurrences of E in that windows, you'll find 4 E's. Did you get it? – Moh Apr 28 '15 at 8:26
  • Given the window size is 2 and if i consider all the occurrences of A, I see only "E A" in the corpus that occurs with A? at 4 and 5 indeces – samsamara Apr 28 '15 at 8:54
  • @KillBill: I've said the window size is 2 of either size, I mean the window is [-2,+2]. Is it clear now? – Moh Apr 28 '15 at 9:07
  • 1
    very helpful explanation – Domenico M. Nov 16 '17 at 13:24

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.