What statistical methods are used to test whether a corpus of symbols is linguistic?

In their 2009 paper on the Indus Script, Rao et al. describe a test for deciding whether a corpus of symbols is in fact a collection of texts in some language. Over-simplifying a bit, the approach they follow can be summarized as follows. They begin by restricting their attention to a given number of most frequently occurring symbols in the corpus, and calculate the conditional entropy of pairs of symbols for this subset. (Conditional entropy in this context quantifies how well one can predict the next symbol, given only the current symbol. Low conditional entropy means the current symbol is a good predictor of the next symbol.) They then plot the conditional entropy against the number of symbols included, and argue that symbol corpora that are linguistic in origin result in similar plots, while non-linguistic symbol systems result in plots that are very different from the plots for linguistic ones.

The approach of Rao et al. and the justifications they gave have been strongly criticized by various authors and a heated discussion followed (there were even rebuttals of rebuttals of rebuttals).

I am curious about other approaches to this fundamental problem. What are some of the specific methods proposed for testing whether a given corpus of symbols is in fact a collection of texts in some language?

• what do you mean by something "is linguistic"? Sep 28, 2011 at 8:17
• Since you are looking for an alternative to the usual method could you consider including the briefest introduction to what the usual method is? Sep 28, 2011 at 8:59
• @LouisRhys, I meant to ask about methods that can be used to test whether a given corpus of symbols is a collection of texts in some language. The Rao et al. paper talks about linguistic vs. non-linguistic symbol systems, and I had assumed the usage is standard.
– Azo
Sep 28, 2011 at 14:37
• @hippietrail, the method proposed by Rao et al. was strongly criticized by the blog posts I linked to (and others), which is the main reason I'm looking for alternatives. I will edit the post to add a brief description.
– Azo
Sep 28, 2011 at 14:37
• @hippietrail, I added a description of the approach of Rao et al.
– Azo
Sep 28, 2011 at 18:14