4

A good QA on StackOverflow led me to this great read about Unicode, The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!). It helped me understand the background of why the UTF-8 encoding is so needlessly (so I thought) complicated. Instead, UTF-8 elegantly solves the problem of backwards compatibility, minimum memory usage for the most common cases, and future-proofing (to a reasonable extent).

In summary, Unicode simply assigns numbers to all possible characters in the world, allowing room for more in the future. An 'a' is different than an 'A', but an 'a' in one font is not different from an 'a' in another font, or an 'a' in a different size. UTF-8, on the other hand, is an encoding scheme where Unicode meets implementation in real-world computer systems.

This got me to thinking about Unicode for languages. In theory, to a layman such as myself, all words encapsulate the knowledge of objects and ideas. Just as an 'a' is an 'a' no matter what font, the word 'tree' encapsulates the concept of a tree, no matter the language. But wait, 'tree' is the English encoding of the concept of a tree, isn't it? In other encodings (aka languages) it might be the equivalent of 'tall bush', or 'plant with thick bark', or 'leafy sky scraper'... I think you get the point. :)

It seems, with considerable and united effort, a "Unicode" for languages could be developed. Heck, it could probably be captured in Unicode itself, since it allows for future additions. And with this, translation problems could, eventually, be a relic of the past.

Is there a "Unicode" of languages, that either exists now or is in development?

19
  • 1
    1. Words are open-class, i.e. new words get created all the time, and it would be a painful to update standards (which are supposed to change infrequently). 2. Look up WordNet and Wiktionary, the two large-scale projects that link words senses across languages. – prash Jul 26 '15 at 21:13
  • 1
    Another problem is that it's not easy to decide what is a word and what isn't. Eg when there are related forms such as 'run, ran, running, runner, runny, runs' etc it can be hard to decide whether these are all forms of one word, are each a separate word, or somehere in between. So would there be a 'word unicode' character for each on of these? Other languages might not have some of these, especially the grammatical variants such as past tense forms of verbs (there are many languages without past tense verb forms). – Gaston Ümlaut Jul 26 '15 at 23:16
  • 3
    However, an approach that might come closest to what you're thinking is Natural Semantic Metalanguage. – Gaston Ümlaut Jul 26 '15 at 23:17
  • @prash Sounds like a problem of scope, which is a solvable problem in engineering. You do what's feasible for the first release, and do more the next release, and so on, until eventually you get a pretty stable release that you don't have to update much. – CivFan Jul 27 '15 at 0:34
  • 1
    Bringing all human alphabets under one system is especially easy, because humans invented the alphabet just once (at least according to Gelb's classic book on writing systems). The various alphabetic systems now in use are closely related because historically they arose from a single progenitor (Phoenician or Greek). However, we didn't invent the dictionary just once. – Greg Lee Jul 27 '15 at 1:03
7

To once and for all codify all of human language to enable the unambiguous cross-linguistic, cross-cultural and cross-temporal dissemination of thoughts and knowledge has been a dream for thousands of years. Its earliest origins, at least in Europe, date back at least to Raimundus Lullus (1232—1315); the big names in this story include the likes of Bishop Wilkins, author of An Essay towards a Real Character and a Philosophical Language, Gottfried Wilhelm Leibniz, and countless inventors of constructed languages around the turn of the 19th/20th c such as, most famously, L. L. Zamenhof, the inventor of Esperanto.

Much has been written and said about these subjects; a very comprehensive, analytic, entertaining and educating treatments of the subjects can be found in Umberto Eco's La Ricerca della Lingua Perfetta nella Cultura Europea (The Search for the Perfect Language in the European Culture).

I do not think this is going to work the way that many inventors of such schemes dreamed it could be, and the reasons are manifold. One difficulty that is easy to see is that 'words' do not denote constant, definite concepts—not across languages and not even within a single language across sociolinguistic domains.

If you absolutely want to pursue that effort anyway, may i recommend a healthy amount of reading; a few of the subjects that should be touched upon are the question about the reality of species, the demonstrated impossibility to represent 'the world' or 'human knowledge' as a hierarchical system (notwithstanding how many times it has been tried over again), expression of color words in languages, discussions about the merits and limits of efforts such as WordNet.

As a very straightforward point to start a serious implementation of your idea, may i suggest the following: Assuming that Wikipedia is a near-perfect representation of human knowledge (is it?), we may stipulate that all Wikipedia HTTP addresses that point to an article automatically become 'words' or 'codepoints' in the Universal Word Encoding (technically, this may be achieved e.g. by calculating hash codes from the addresses and translate those to some kind of numbers of some kind of pronounceable phantasy words). Then, let's trace all the translation links of all the words. Wherever there is a strong clustering of articles, we may assume that a given 'word' has a comparable meaning across many languages; we may then again identify that cluster by quoting one of its members (discarding all the synonyms).

I hope you can see where this goes. It's inevitably going to be a huge amount of pretty hard work. Basically, you'd have to absorb a major part of what's written in the world's encyclopedias.

Update @CivFan—"Pointing to the failings of existing solutions does not prove that no good solution is feasible."

In a way you're right and of course you're free to try and go boldly where no man has gone before. On the other hand, it would be foolish to ignore failed attempts and understand why those have failed. Ottmar Mergenthaler, inventor of the Linotype succeeded where hundreds of inventors had failed, namely, to devise a system to speed typesetting by mechanical means; this was, among other things, due to his deep understanding of what can be done and what had been shown to be not viable.

I have one more example where one should look for failed solutions, and this one is closely related to Unicode.

When you look at how Unicode is set up, you will see it is, in essence, just a linear enumeration of known characters / smallest units of writing gathered from the world's writing systems. Now, these characters are indeed collected into blocks that bear such names as 'Cyrillic', 'Greek', 'Arabic' and so on; however, as a rule of thumb, whenever you look at what a given language needs to get written, you'll typically find that it needs characters from more than a single Unicode block; also, the way that words are normally sorted in that language will most of the time only partially match the way those characters are numerically arranged in Unicode. These properties have been well known to the Unicode Consortium right from the outset, and they are not seen as shortcomings; rather, it is the general understanding that the convenient construction of US-ASCII (all 26 letters in right order; conversion between upper and lower case doable by simple bit switching) does not scale when you construct a code for multiple languages and multiple writing systems. This is relevant in so far as you are basing your ideas explicitly on Unicode; now we can quickly agree that the fact that Unicode does (successfully) exist, but a 'semantic Unicode' does notmight be indicative that the latter one is significantly harder to do than the former.

Part of the reason why Unicode does not provide, say, bit-swapping case mappings across languages and scripts a la US-ASCII is that Unicode has a linear code space, and there's only so much worthwhile complexity you can build into that single dimension. So, essentially, what this tells us is that if Unicode could not (and should not) make use of ingenuous arithmetic tricks to implement the ability to map lower case letters to upper case letters, then that much fuzzier realm of 'semantics' should be much harder to codify using a 'speaking code'. In essence, this means that whatever you come up with will be most likely either something that is not linearly ordered (but more network like or whatever), or, if it is linearly ordered, be a haphazard enumeration of concepts. This need not even be useless, but i wonder whether it is what you imagine it could be.

3
  • Start with Frawley's Linguistic Semantics and Foley's Anthropological Linguistics. And then go to Lakoff and Johnson's Metaphors We Live By, and possibly their Philosophy in the Flesh. The first two books are a survey of variation in human lexical semantics, and lists of basic perceptual categories; the second two deal with metaphor, which is the source of almost all our meanings, and how projections of the human body are inside almost all our metaphors. – jlawler Jul 27 '15 at 13:24
  • 2
    Very much to the point. The moment you have found the universal words for 'duck' and 'row' and a few other lexical items, you still have to 'get a lot of ducks into the row' in order to get the ducks into a row. IOW, language is much more than just a repository of well-known nouns (and even the nouns are already fiendishly difficult to categorize and unambiguously identify). – flow Jul 27 '15 at 13:39
  • Agreed, looking to the failings of existing solutions and past attempts is critical if the problem is to ever be solved. I of course suspected there was no such Unicode for words, otherwise I probably would have found it already. But in that case, I was hoping for answers that expound on past attempts and why they failed, and to see the problems the next attempts would have to avoid or solve. There are many great answers already, but I think this one at the moment does the best of referencing past attempts and suggesting possible ways forward. I wish I could give more than one answer credit. – CivFan Jul 27 '15 at 18:07
2

Words are spelled in alphabetic systems with letters, and the number of different letters required for all the words of human languages is finite and rather small, so you can assign a small number to each letter. This makes Unicode feasible. But human languages are made of sentences, and the number of different sentences required for all human languages (or any one of them) is infinite. This makes a Unicode for languages impossible in principle, since you could never assign a small finite number to each possible sentence.

7
  • Yes, language has infinite capabilities, but in practice the scope for encoding words is large but finite. – CivFan Jul 27 '15 at 0:46
  • 1
    @CivFan No, it is not finite: no two languages divide the conceptual universe into words the same way; and no one language (unless it is completely dead) continues to divide the conceptual universe into words the same way from year to year. You would have to have a semantic partition of the conceptual universe which everyone could agree on. And then the conceptual universe itself keeps changing . . . – StoneyB on hiatus Jul 27 '15 at 1:37
  • @StoneyB Sounds like a scope problem. Implementation of theory doesn't have to cover all possible cases to be useful. Standardizing words would actually make the problem of the evolving meaning of words easier to handle. It's a problem that exists regardless, right? – CivFan Jul 27 '15 at 2:19
  • 2
    @CivFan Evolving meaning is not an obnoxious problem to be overcome but a powerful feature to be exploited. – StoneyB on hiatus Jul 27 '15 at 18:52
  • 1
    @CivFan again I point to Natural Semantic Metalanguage as an approach that may show one way to do this, and probably the most nearly practical. – Gaston Ümlaut Jul 27 '15 at 22:26
2

There is an obvious difference known between words and letters that should make Unicode systems feasible for letters, but does not apply to words. So far as I've noticed no one has mentioned it, so despite my having already offered one answer, I think I should mention the obvious.

Alphabetic writing systems have letters which correspond roughly to phonemes. You could think of them as primitive and traditional phonemic theories. Here are two things we know about the phoneme systems of the various human languages: (1) Each language has a very small number of phonemes -- around 50-200 -- which are very stable over time, and (2) the phoneme systems of the various languages are very similar, making practical phonetic alphabets like IPA and feature systems like that of SPE (with about 17 binary features, giving 2^17 possible sound segments).

Are the words of human languages anything like this, being very small in number and limited (and fairly well understood) in their reference? No. Not at all.

2

Unicode (within and beyond the BMP) has nearly 75,000 code points for CJK (Chinese Japanese Korean). The vast majority of these characters are actual words (most modern Chinese words are made of more than a single ideogram, yet most Kanji/Hanzi characters have meanings equivalent to our words).

More generally Unicode encodes many (at least partially) ideographic writings (many of them from ancient languages: e.g. Hittites, Cretan and Egyptian hieroglyphs, etc...).

So Unicode already has a lot of characters for "words".

Having one character per word could seem a nice idea but there are a few hurdles in the way of course. If you have one character for each word, you need:

  • Readers and writers with a huge brain to remember them all. Most Chinese people get away with under 5000 characters though.

  • A low morpheme-per-word ratio.

  • Huge font files.
  • A sophisticated input method for your keyboard (if you don't want to end up with large keyboards such as this famous example for Chinese).

From Wikipedia - https://en.wikipedia.org/wiki/Chinese_input_methods_for_computers

So basically, a Unicode for words, as you put it, is probably for the time being a regression for many synthetic languages. After all, before the advent of computers, the Phoenician alphabet was hailed as an important progress.

This might change with the generalisation of Text To Speech and Voice Recognition technologies. But we're not there yet.

4
  • The fact that Unicode encodes CJK makes me think there's some hope for a Unicode of words, with limited scope. Assuming the Unicode for words problem could be solved, making a system to type them would be an interesting path that would lead from it. Seeing how the problem of typing the large number of characters is handled now is interesting. – CivFan Jul 27 '15 at 18:11
  • 1
    @CivFan - The input method software for mandarin uses pin-yin (i.e. so back to alphabetical input) with prediction (a little bit like on your phone). So you can't have it both way: if the number of keys is lower than the number of words, then you have to use combinations of them (so you still need to learn some kind of alphabet - IPA?). An interesting case is that of Stephen Hawking's voice synthesiser driven by internal cheek movements (also with word prediction). All this of course is not necessary any more when our input device is directly plugged into our cortex... in a few years time. – Alain Pannetier Jul 27 '15 at 18:31
  • - "not all Chinese words are made of a single ideogram of course, but most Kanji/Hanzi characters are equivalent to our words" the majority of words in modern Chinese are multi-character (languagelog.ldc.upenn.edu/nll/?p=3330) - "Most Chinese people get away with under 3000 characters though." That's probably a bit low, unless we're averaging in illiterate people which makes the number less meaningful. (See zhidao.baidu.com/question/…, zhidao.baidu.com/question/… etc) – MHG Jul 31 '15 at 8:05
  • @Xiu. Thanks. Corrected according to your review and sources. – Alain Pannetier Jul 31 '15 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.