2

I would like to construct a normalization program for text containing diacritics or special symbols. For some languages that I am familiar with, I can obtain a canonical form easily;

German:

ä -> ae 
ö -> oe
ü -> ue

French:

é -> e
è -> e
î -> i
à -> a
â -> a
ô -> o

My questions are:

  1. Does such a mapping exist for these languages: Spanish, Italian, Portuguese, Swedish/Norwegian, Turkish, Czech, etc. (European/Indo-European).
  2. What is the mapping? :)
Improve this question
2
  • Is this just for fun, or does the normalization program have a real-world application that you can tell us about?
    – brass tacks
    Jul 21, 2015 at 20:18
  • @sumelic it is mostly for fun, to be able to play with text documents in many languages -- it would allow me to do analyses on a smaller character set.
    – mitchus
    Jul 22, 2015 at 6:57

1 Answer 1

Reset to default
4

Your two examples both show a different kind of process.

In German, you remove the diacritics and retain the information that they encode. And you do it in a way that is broadly consistent with the conventions of German orthography for removing diacritics.

However, in French, you simply remove the diacritics and any information that may have been encoded by them is lost.

You can do the latter for any language with diacritics. Simply, remove them and use the equivalent plain ASCII characters. You will increase the amount of ambiguity in the text but not beyond what native speakers would be able to cope with. So in Czech ěščřžýáí become escrzyai. You end up with things like pani which could be either paní (Mrs) or páni (gentlemen). This is typically not a problem in context and occurs less often than you'd think.

In the early days of email before unicode, speakers of languages with diacritics outside ASCII often resorted to simply writing in the ASCII equivalents (and many still do). There were also many ad hoc systems for preserving the diacritics such as e^s^c^r^z^y'a'i' (or a" for umlauts) but they never really caught on because they made the text harder to read than removing the diacritics altogether.

Also, remember that no all non-ASCII characters in European languages are simply formed by adding diacritics. For instance, ß in German which would be typically replaced with ss, ø in Danish or ł in Polish.

Algorithms for doing this already exist. See for instance, http://www.siao2.com/2005/02/19/376617.aspx and has been implemented in many contexts in software.

Improve this answer
2
  • Actually, the two processes I showed are not so different, because they correspond to how German- and French- speaking people usually transcribe text when diacritics are not available, for example when they write something in UPPERCASE. While in French the "information lost" is deemed to be sufficiently small to just throw the diacritics away, in German there is an explicit (and widely used) mapping. This is why I am asking if such widely used mappings exist for other languages as well. The link that you gave, for example, does not respect the usual German mapping.
    – mitchus
    Jul 22, 2015 at 14:37
  • Yes, if you're asking about conventional ways of doing this across languages, these are not different. Sorry, I did not read your question that way. I'm not aware of any study that has looked at this. It might be worth a quick exploration but not sure there's that much in it.
    – Dominik Lukes
    Jul 23, 2015 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.