1

I'm doing my own (very small) corpus analysis, including a word frequency list and would like to compare the top, say, 30 words with other such lists. I know that there are things like the GSL/NGSL, the Leipzig-Jakarta list or the Swadesh list. But the latter two are concerned with a different question (stability of words across time) and the former two are intended for study more than for research. Also, my own corpus consists of spoken language (transcripts of videos to be exact) which can be very different from written language.

So my question has three parts:

Is there a collection of word frequency lists (top 100 words would be enough) of a number (>10) of languages anywhere?

Has anyone tried to translate a list of such lists into one common language and readjusted the frequencies?

If the latter exists, how accurately does it approach "the top 100 words spoken by all speakers of all of the included languages" in your opinion? (I realize things get lost in translation quickly, but the lists would be likely to include things like "I/you/we/they", "to say/to come/to go", "home/work/school" which are quite universal as concepts).

| improve this question | |
  • In fact, this are three questions in one. The first applies maybe to open data, or you can get at wikipedia dumps in several languages and compute the top lists yourself. – jk - Reinstate Monica Jul 15 '15 at 13:52
  • 1
    The second and third question are kind of moot: the most frequent words are often overloaded with lots of meanings (e.g. "to go") and idioms and have often dozens of very different translations into other languages. – jk - Reinstate Monica Jul 15 '15 at 14:27
  • Here's a start: ladybird.co.uk/worldwide/key-words – Hugh Jul 15 '15 at 15:07
  • And two more top 1000 here: kyle-p-johnson.com/notebooks – Hugh Jul 15 '15 at 15:15
  • Thank you for the comments so far and sorry for throwing some things together into one pot. Hugh's link is quite useful and also shows one of the problems in this domain - it's not at all easy to compile such lists. My own corpus is a transcript of 5 Arabic YouTube videos totalling around 1200 words and it's taken me around a week to manually tokenize and lemmatize everything. Doing so in an automated fashion is (mostly) out of the question for Arabic so far. Hugh's list has Greek and Latin, for English there are actually lots of lists and German seems well-covered, too. Other languages, hmm. – Sixtyfive Jul 16 '15 at 14:30
3

  1. http://www.ezglot.com/most-frequently-used-words.php

  2. the above site does this something like this for common words, and homepage has a list of few most widely understood.

| improve this answer | |
  • Thank you for that link anonymous! If I could, I'd upvote it. The page is interesting for the large number of languages (around 90). The three languages I can say something about are German, English and Arabic. In all three, the top 100 lists are far from perfect, but they're not bad either. It would be nice if offered a detailed list of the corpora used as well as an in-depth explanation of their counting algorithm (which they claimed took months to develop). Comparing to (N)GSL, the English count seems really good. Arabic needs work and German seems really bad. Still, thank you for the link! – Sixtyfive Jul 20 '15 at 9:43
1

Word frequency is already subjective within one language; across languages it's completely arbitrary

The most frequent words are function words. But a function word in one language may be expressed by morphology or syntax in another, or by another part of speech.

So your results will be decided by what you count as a word, and which languages you include, and how they are weighted.

Even worse than arbitrary but conscious decisions is making conclusions without realising the methodology over-counts words from languages with a certain typology.

| improve this answer | |
  • 1
    Wow, talk about reviving the dead - I ended up being able to finish my Bachelor's thesis and have moved on to other things since then, but thank you for the answer, Adam! – Sixtyfive May 26 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.