10

In their 2009 paper on the Indus Script, Rao et al. describe a test for deciding whether a corpus of symbols is in fact a collection of texts in some language. Over-simplifying a bit, the approach they follow can be summarized as follows. They begin by restricting their attention to a given number of most frequently occurring symbols in the corpus, and calculate the conditional entropy of pairs of symbols for this subset. (Conditional entropy in this context quantifies how well one can predict the next symbol, given only the current symbol. Low conditional entropy means the current symbol is a good predictor of the next symbol.) They then plot the conditional entropy against the number of symbols included, and argue that symbol corpora that are linguistic in origin result in similar plots, while non-linguistic symbol systems result in plots that are very different from the plots for linguistic ones.

The approach of Rao et al. and the justifications they gave have been strongly criticized by various authors and a heated discussion followed (there were even rebuttals of rebuttals of rebuttals).

I am curious about other approaches to this fundamental problem. What are some of the specific methods proposed for testing whether a given corpus of symbols is in fact a collection of texts in some language?

Improve this question
6
  • what do you mean by something "is linguistic"?
    – Louis Rhys
    Sep 28, 2011 at 8:17
  • Since you are looking for an alternative to the usual method could you consider including the briefest introduction to what the usual method is?
    – hippietrail
    Sep 28, 2011 at 8:59
  • @LouisRhys, I meant to ask about methods that can be used to test whether a given corpus of symbols is a collection of texts in some language. The Rao et al. paper talks about linguistic vs. non-linguistic symbol systems, and I had assumed the usage is standard.
    – Azo
    Sep 28, 2011 at 14:37
  • @hippietrail, the method proposed by Rao et al. was strongly criticized by the blog posts I linked to (and others), which is the main reason I'm looking for alternatives. I will edit the post to add a brief description.
    – Azo
    Sep 28, 2011 at 14:37
  • 1
    @hippietrail, I added a description of the approach of Rao et al.
    – Azo
    Sep 28, 2011 at 18:14

1 Answer 1

Reset to default
5

Based on the article, the authors used what is known as a Markov model. This is actually the departure for discovering the linguistic value of a text, where the normal methods are those that are used in traditional Cryptanalysis, which is the attempt to discover the meaning (or linguistic value) of a text of seemingly random symbols or systems. More can be found in a basic introduction to Cryptography, and here: Wikipedia - Cryptography (#Cryptanalysis)

Improve this answer
1
  • 2
    Thanks for the link, Mark. I'm looking for pointers to specific tests for whether a symbol system/corpus is really a writing system. Rao et al. do use Markov chains, but they extract a quantity (the conditional entropy) out of the chains they fit, and describe a specific test (although, admittedly, a vague one). If you know of other specific approaches based on Markov models, I'd be interested in learning about them.
    – Azo
    Sep 28, 2011 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.