How to count the number of unique words (lemmas) in a Russian text?

Question

Here's the problem: I have a large deck of Anki flashcards which I memorize. I would like to know how many unique Russian words (lemmas) I have in my entire collection.

What I need is some kind of lemmatizer or word counter or morphological dictionary that can tell me which words are different, and which are the same.

For example these words should be counted as the same: делать and делаю, хороший and хорошего, ме́сто and места́. However, здорово and здоровый should be different words.

Anyone know if this is a solved problem?

I imagine a simple computer program would work like this:

1. Determine morphological root of the word (здоровому -> здоровый, здорово -> здоровый). This seems to be solved by https://morfologija.ru
2. Determine whether the derivation has the same meaning as the root. (здоровому is equal to здоровый, but здорово is NOT equal to здоровый because they have different entries in a dictionary)
3. Count number of unique lemmas.

Here are some resources I have gathered online, but I am not sure how to use them.

• Yandex MyStem https://yandex.ru/dev/mystem This seems to help find the root of a word, but I don't see any word counters built with this.

• https://morfologija.ru looks like it might help to find the "root" or lemma of a word. It seems to provide derivational root. However it doesn't provide an API and it is still not clear to me what to do when you have multiple listed "roots".

Answer 1

There exist quite a few Russian lemmatizers (pymorphy2 probably being the most popular), but I'm not sure if any of them can do word/lemma counting. So you will probably have to write a word counting script yourself. It's a much simpler task than lemmatization.

Regardless of which approach you take, you're bound to run into ambiguity at some point. E.g. the word «пошли» can be lemmatized as «пойти», «послать» or «пошлить». There is no easy way around this. You need more context to choose the correct lemma.