Turn Penn Treebank into simpler POS tags

Question

I'm working on some code for an open source package to analyze dialogic classroom transcripts. I came across an interesting article that calculates a formality measure that I wanted to try out (LINK) as it may help my field (literacy) understand what explicitness of teacher talk means.

Heylighen, F., & Dewaele, J.M. (2002). Variation in the contextuality of language: 
     An empirical measure. Context in Context, Special issue of Foundations of 
     Science, 7 (3), 293–340.

The formula for the statistic is fairly straight forward (p. 309):

F = (noun frequency + adjective freq.  +  preposition  freq.  +  article  freq.  –  pronoun freq. – verb freq. – adverb freq. – interjection freq. + 100)/2

There happens to be a part of speech tagegr in the program I use (R) that is over 95% accurate on tagging POS. So I first run the POS tagger on the transcript and get counts for parts of speech in a matrix form. The problem is that Heylighen & Dewaele use a rather simple POS code as seen in the formula above where as the POS tagger I use, uses Penn Treebank coding (LINK):

   Tag  Description                             
1  CC   Coordinating conjunction                
2  CD   Cardinal number                         
3  DT   Determiner                              
4  EX   Existential there                       
5  FW   Foreign word                            
6  IN   Preposition or subordinating conjunction
7  JJ   Adjective                               
8  JJR  Adjective, comparative                  
9  JJS  Adjective, superlative                  
10 LS   List item marker                        
11 MD   Modal                                   
12 NN   Noun, singular or mass                  
13 NNS  Noun, plural                            
14 NNP  Proper noun, singular                   
15 NNPS Proper noun, plural                     
16 PDT  Predeterminer                           
17 POS  Possessive ending                       
18 PRP  Personal pronoun                        
19 PRP$ Possessive pronoun                      
20 RB   Adverb                                  
21 RBR  Adverb, comparative                     
22 RBS  Adverb, superlative                     
23 RP   Particle                                
24 SYM  Symbol                                  
25 TO   to                                      
26 UH   Interjection                            
27 VB   Verb, base form                         
28 VBD  Verb, past tense                        
29 VBG  Verb, gerund or present participle      
30 VBN  Verb, past participle                   
31 VBP  Verb, non-3rd person singular present   
32 VBZ  Verb, 3rd person singular present       
33 WDT  Wh-determiner                           
34 WP   Wh-pronoun                              
35 WP$  Possessive wh-pronoun                   
36 WRB  Wh-adverb  

That means I have to turn the Penn Treebank coding into Heylighen & Dewaele coding. Here is my attempt to do that:

DF1 <- data.frame( 
    noun = rowSums(X[, names(X) %in% c("NN", "NNS", "NNP", "NNPS", 
        "POS", "JI", "JK", "CD")]), 
    verb = rowSums(X[, names(X) %in% c("MD", "VB", "VBD", "VBG", 
        "VBN", "VBP", "VBZ", "JI", "JK")]),  
    adverb = rowSums(X[, names(X) %in% c("RB", "RBR", "RBS", "WRB", 
        "JI", "JK")]),  
    pronoun = rowSums(X[, names(X) %in% c("PRP", "PRP.", "WDT", "WP", 
        "WP.", "JI", "JK", "EX")]),  
    prep = rowSums(X[, names(X) %in% c("IN", "RP", "TO", "JI", "JK")]),  
    adj = rowSums(X[, names(X) %in% c("CD", "DT", "JJ", "JJR", "JJS", 
        "JI", "JK")]),  
    interj = rowSums(X[, names(X) %in% c("UH", "JI", "JK")]))

So you can see I combine the "NN", "NNS", "NNP", "NNPS", "POS", "JI", "JK", "CD" tags into nouns. You may notice there's no articles in my code, it seems that Penn Treebook doesn't have an articles class but may classify them as other codes (I think maybe deterministic; this may be problematic in that is appears deterministic is a mix of adjectives and articles which are on opposites in Heylighen & Dewaele's measure). The problem is I run the code and I sometimes get negative numbers (according to Heylighen & Dewaele this is not possible) . This isn't because of the code or math but because I haven't classified the Penn Treebank into Heylighen & Dewaele's codes correctly. I am not a linguist and my attempts to understand this problem have proven to be well above my knowledge or research capabilities.

I am asking for help in correctly converting these codes or letting me know if what I'm attempting isn't feasible.

Answer 1

Penn Treebank does have a POS tag for articles — they're determiners, DT, and probably shouldn't be mapped to adjectives as they are in your code. I wonder if that could be the source of your troubles.

It also seems that you're mapping some PTB tags (e.g. CD) to more than one coarse-grained tag. Could that be messing up some of the counts?

Also, Stanford NLP has developed a universal POS tag set and mappings from 25 treebanks' tagsets¹, often generalising over finer-grained tagsets. You can compare the mapping from the PTB to the mapping scheme you have developed.

---

^{1: Petrov S., Das, D., McDonald R. "A Universal Part-of-Speech Tagset" http://arxiv.org/pdf/1104.2086v1.pdf}