I was looking for lists of words (preferably txt files) in various languages I was looking for just lists of words without any additional data spanning as many languages as possible. Thanks

  • 1
    What should those lists be good for? Please give more details. I assume, without additional data also means "without translation"? Apr 3 '18 at 10:05
  • I was looking for plain text lists of words in various languages, for example a aardvark abacus ... Apr 13 '18 at 5:26
  • Only lemmas, or a list of wordforms (including inflected forms)? Apr 13 '18 at 10:06
  • List of wordforms Apr 13 '18 at 12:52

FAIR have trained word embedding models for hundreds of languages, with the side effect that they have compiled words lists.

Wikipedia only, 300+ languages

fasttext.cc/docs/en/crawl-vectors.html / github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md

Wikipedia + Common Crawl, 157 languages

fasttext.cc/docs/en/pretrained-vectors.html / github.com/facebookresearch/fastText/blob/master/docs/crawl-vectors.md

As noted under Format, the words are sorted in order of frequency. They provide Python code, if you do not care about the vector representation then you could use bash to split on space and take only the first column.)

To download them all, get download.sh from the ftio package:

for (( i=1; i<=$#; i++ )); do
    wget -c "https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.${!i}.zip"

# For example:
# ./download.sh bg el ka hy ru fa es fr de it pt ar tr pl ko

# If stopped it will not re-start automatically, but if re-started it will continue from where it stopped.

# Tested on macOS Sierra only.

# Remember to check if there is enough disk space - they are about 4GB each.

(For Wikipedia + Common Crawl, change wiki.${!i}.zip to word-vectors-v2/cc.${!i}.300.vec.gz.)

You can also try Open Data SE.

  • Those are huge files. What is the format of the 'embedding'?
    – amI
    Apr 5 '18 at 21:45
  • @aml Every line is <word> <vector>, where <vector> is 300 numbers separated by spaces. Apr 6 '18 at 6:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.