# Turn Penn Treebank into simpler POS tags

I'm working on some code for an open source package to analyze dialogic classroom transcripts. I came across an interesting article that calculates a formality measure that I wanted to try out (LINK) as it may help my field (literacy) understand what explicitness of teacher talk means.

``````Heylighen, F., & Dewaele, J.M. (2002). Variation in the contextuality of language:
An empirical measure. Context in Context, Special issue of Foundations of
Science, 7 (3), 293–340.
``````

The formula for the statistic is fairly straight forward (p. 309):

``````F = (noun frequency + adjective freq.  +  preposition  freq.  +  article  freq.  –  pronoun freq. – verb freq. – adverb freq. – interjection freq. + 100)/2
``````

There happens to be a part of speech tagegr in the program I use (R) that is over 95% accurate on tagging POS. So I first run the POS tagger on the transcript and get counts for parts of speech in a matrix form. The problem is that Heylighen & Dewaele use a rather simple POS code as seen in the formula above where as the POS tagger I use, uses Penn Treebank coding (LINK):

``````   Tag  Description
1  CC   Coordinating conjunction
2  CD   Cardinal number
3  DT   Determiner
4  EX   Existential there
5  FW   Foreign word
6  IN   Preposition or subordinating conjunction
10 LS   List item marker
11 MD   Modal
12 NN   Noun, singular or mass
13 NNS  Noun, plural
14 NNP  Proper noun, singular
15 NNPS Proper noun, plural
16 PDT  Predeterminer
17 POS  Possessive ending
18 PRP  Personal pronoun
19 PRP\$ Possessive pronoun
23 RP   Particle
24 SYM  Symbol
25 TO   to
26 UH   Interjection
27 VB   Verb, base form
28 VBD  Verb, past tense
29 VBG  Verb, gerund or present participle
30 VBN  Verb, past participle
31 VBP  Verb, non-3rd person singular present
32 VBZ  Verb, 3rd person singular present
33 WDT  Wh-determiner
34 WP   Wh-pronoun
35 WP\$  Possessive wh-pronoun
``````

That means I have to turn the Penn Treebank coding into Heylighen & Dewaele coding. Here is my attempt to do that:

``````DF1 <- data.frame(
noun = rowSums(X[, names(X) %in% c("NN", "NNS", "NNP", "NNPS",
"POS", "JI", "JK", "CD")]),
verb = rowSums(X[, names(X) %in% c("MD", "VB", "VBD", "VBG",
"VBN", "VBP", "VBZ", "JI", "JK")]),
adverb = rowSums(X[, names(X) %in% c("RB", "RBR", "RBS", "WRB",
"JI", "JK")]),
pronoun = rowSums(X[, names(X) %in% c("PRP", "PRP.", "WDT", "WP",
"WP.", "JI", "JK", "EX")]),
prep = rowSums(X[, names(X) %in% c("IN", "RP", "TO", "JI", "JK")]),
adj = rowSums(X[, names(X) %in% c("CD", "DT", "JJ", "JJR", "JJS",
"JI", "JK")]),
interj = rowSums(X[, names(X) %in% c("UH", "JI", "JK")]))
``````

So you can see I combine the "NN", "NNS", "NNP", "NNPS", "POS", "JI", "JK", "CD" tags into nouns. You may notice there's no articles in my code, it seems that Penn Treebook doesn't have an articles class but may classify them as other codes (I think maybe deterministic; this may be problematic in that is appears deterministic is a mix of adjectives and articles which are on opposites in Heylighen & Dewaele's measure). The problem is I run the code and I sometimes get negative numbers (according to Heylighen & Dewaele this is not possible) . This isn't because of the code or math but because I haven't classified the Penn Treebank into Heylighen & Dewaele's codes correctly. I am not a linguist and my attempts to understand this problem have proven to be well above my knowledge or research capabilities.

I am asking for help in correctly converting these codes or letting me know if what I'm attempting isn't feasible.

• Where do 'JI' or 'JK' or 'POS' come from? Those aren't in the PTB tagset. Jul 15, 2012 at 1:19
• Good question, they're merely place holders for the code and are later dropped (JI and JK) but POS is possessive pronoun. Jul 15, 2012 at 1:52
• Forgot about `POS` :P I've been working with Penn Chinese Treebank's tagset too much... Jul 15, 2012 at 2:51

Penn Treebank does have a POS tag for articles — they're determiners, `DT`, and probably shouldn't be mapped to adjectives as they are in your code. I wonder if that could be the source of your troubles.
It also seems that you're mapping some PTB tags (e.g. `CD`) to more than one coarse-grained tag. Could that be messing up some of the counts?