Do standard corpus analysis tools like AntConc and Wordsmith work for foreign languages?

Question

Can I use tools like AntConc and Wordsmith and produce decent results when analyzing languages like French, German or Spanish?

Would it deal with special symbols like "à", "ü", "ß", "ç" and similar letters not present in English?

Would it deal with morphological diversity, and when counting words, would it know that the pair "parle" and "parlons", or, "Mann" and "Männer" have actually the same root each?

What other problems would I need to deal with?

Answer 1

I believe Wordsmith 3+ works fine, and in the case of Antconc special characters will sometimes mess when using Concordancer. for the second question you can use the lemmatizer in Antconc.

Answer 2

These are two separate questions.

1. Will AntConc and Wordsmith deal with characters not present in English (sometimes referred to as non-ASCII)? This is really a question about encoding of text (not language). AntConc supports unicode UTF-8 which means it should deal with any script. WordSmith only supports a limited subset which means that texts in non-Latin scripts will have to be converted. But none of the examples you give will present any problems. This is a relatively easy issue to resolve depending on the encoding of the texts you provide.

2. The question about dealing with morphology is one of lemmas (which are essentialy word stems without any inflection). Both Wordsmith and AntConc support lemmas but you will have to provide a lemma list for each language. There are guides online on how to do it and a pre-made lemma list for English also exists. A good source for data for generating a lemma list for other languages are the many open source spell checks - but it will require work and coding to create those.

Answer 3

AntConc works very well with non-English languages, including European languages, Japanese, Korean, Chinese, as well as right-to-left languages such as Arabic and Hebrew.