4

Searching "the number of languages" shows about ~7,000.

However, Google Translate only has ~100 languages listed.

enter image description here

That makes me wonder if the languages have a lot of overlapping/duplicated elements.

I am interested to know if languages can instead be broken down into "characters sets". For example, English would have a-zA-Z0-9(). -+?!..., while spanish would have a-zñA-Z0-9(). -+?¿!¡..., and french would have a-zç....

Unicode has everything. But I'm wondering if there is a list of either (a) the character sets per language (just the 100 languages on Google Translate, since all ~7,000 would be a monumental task), or a more broad character set classification that groups them somehow into hierarchies, so for example there are the "lowercase letters", which belong in english, french, latin, etc., then there is ¿¡ which spanish adds to the lowercase letters, etc. Basically wondering what sort of classification systems exist for language symbols other than unicode. Or if not that, a database or website that lists the characters for different language so it doesn't have to be aggregated from a bunch of places.

Improve this question
7
  • 1
    Many languages "overlap" in their writing systems; you might want to explore Omniglot for more info on writing systems and languages. – Jeff Zeitlin Aug 27 '18 at 18:12
  • Look up the words alphabet, abjad, and abugida to see some of the variations possible. – jlawler Aug 27 '18 at 20:27
  • Are you asking if there is a list of "the letters used by language X", for some set of languages or all languages? – user6726 Aug 27 '18 at 20:42
  • Yes I am asking if there are a list of letters for each language (or for some of them at least to get started. For example, the french orthography doesn't list the ç as one of it's "characters", it only says "french letters use a few diacritical marks", and you have to figure out which ones are used where. I would like a list that lists all combinations of characters / diacritical marks / etc., or at least the system that says how to combine them. So it's all in one place. – Lance Pollard Aug 27 '18 at 20:52
  • 7
    Please don't assume that writing systems have any essential connection with languages. Languages may have zero, one, or several writing systems associated with them, and in almost all cases the choice of writing system is an accident of history, politics, or religion. – Colin Fine Aug 27 '18 at 22:08
4

Richard Ishida has a ton of online tools related to unicode and orthography. The most useful one for your purposes is probaby the Character Usage Lookup tool. Not a database, but you might be able to figure out a way to get what you want by looking at his source code, or asking him.

Improve this answer
6
  • Oh yes this is pretty much exactly what I'm looking for. Thank you!! – Lance Pollard Aug 27 '18 at 22:44
  • It has some oddities. For ü and ö it suggests Dutch, but Dutch only has it for diaeresis, and if it counts diaeresis then it should suggest English too. – Adam Bittlingmayer Aug 28 '18 at 7:22
  • It is also missing « for some languages, like Croatian and Turkish. – Adam Bittlingmayer Aug 28 '18 at 7:37
  • And it only tells us what an orthography has in addition to ASCII, but not the ASCII characters it does not have. So there is no way to discover that Serbian (Latin) generally excludes x, w or y. – Adam Bittlingmayer Aug 28 '18 at 7:49
  • @A.M.Bittlingmayer: The use of the trema to mark dieresis is (if I understand correctly) obligatory in standard Dutch, but not obligatory--and in fact dispreferred by most style guides--in English. – brass tacks Aug 28 '18 at 8:59
3

There is the CLDR - Unicode Common Locale Data Repository that contains beside locale settings also additional information on many languages, e.g., the exact character set used.

Improve this answer
1

These rules are a bit subjective.

For example you did not include ü in Spanish although ü probably occurs as much in formal Spanish corpora as w. You did not include é in English, although it occurs in a few recent loanwords just like w occurs in Spanish. Standard German has ß, but Swiss Standard German does not.

So for most applications I would avoid the prescriptivist rules-based approaches and tend towards the descriptivist statistical approaches like those used in language identification.

Then it becomes a question of choosing the 1) corpus (eg newspapers, or Wikipedia), 2) pre-processing and 3) a frequency threshold. That scales to many languages and will force you to be more conscious of the subjective choices.

Not a direct answer to your question, but Unicode has character blocks for scripts, which is useful for some languages which are written in one unique script, but as others noted, scripts and orthographies do not map to languages 1:1.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.