4

Disclaimer: I am a software developer, not a computational linguist and I am not super familiar (though I am mildly familiar) with the field. Happy to learn and be corrected though!

My use-case:

I am trying to analyze a text in Chinese, and have grammar rules printed out (in English) based on what appears in the text. These would be simple sentences describing the grammar rule in question.

I have explored several statistics-based parsers (Stanford, SyntaxNet, and SpaCy) and I am looking into how rule-based processing works and I'm somewhat lost - I'm not sure what the right approach is here. It seems that what I want (my guess) is a rule-based approach, and then just mapping that into plain text language - but I don't know what I don't know.

Ideally a solution would be made for Python as that's the language the rest of the application is being done in with the anticipation that most language tools will be available for it.

I am certainly open for using a tool in a different language or even a different approach entirely if that's the best idea.

Here is an example sentence:

人们都穿上了很厚的衣服。

This would generate, for example, this grammar rule:

Verb / Verb Phrase + 了 - Expressing that an action has been completed with "Verb / Verb Phrase + 了"

Due to detecting 了, as long as this character was being used in that particular grammatical way above.

So basically it would look for certain characters, see how they are being used in a sentence, and then spit out a plain text English language rule based on that (presumably based on some mapping of the grammar rules to plain text English that I would write)

My knowledge of computational linguistics is somewhat limited (I am a layman, though I am reading heavily every day) so I am trying to understand the best way forward for this goal.

  • (sorry, ignore the previous comment) I am confused as to what your goal is when you say you want to "generat[e] plain language grammar rules". Can you give an example of input and output you are envisioning? Any parser, whether rule-based or statistics-based, will output structures (is this what you mean by "rule"?); spaCy, for example, gives you 2 structures: constituency and dependency. The 'rules' to derive those structures from text are implicit, given the statistical nature of the parsers. – Rodrigo Jan 8 '19 at 9:44
  • Essentially I want to have a Chinese text, and have a parser analyze this text and when it sees a piece of Chinese that has some grammar rule that applies to it, it generates plain text English language describing the rule. I'll edit my answer in a few minutes with an example. – Andrew Alexander Jan 9 '19 at 1:42
  • 1
    What you mean by 'rule' is a combination of syntactic label (verb) plus some explanation of the label (expressing that...). To label text, you can just use an off-the-shelf parser; no need to build your own. For the explanation bit, I believe all you need is a simple mapping between labels and explanations. When you say "了" expresses "an action [that] has been completed", this to me sounds like some form of participle marking. For example, English parsers would tag "gone" as VBN, i.e. the past participle of "go", so one could map VBZ -> "an action [that] has been completed". – Rodrigo Jan 10 '19 at 10:16
  • Though I would be good at it, I don't understand what are you trying to say. This combination [verb + 了] means something that had happened in the past and is completed now. First add more to that for me to understand. Which one are you asking, the meaning or the mapping? – zixuan Jan 15 '19 at 0:30
  • @Rodrigo - yes. In fact I went with HPSG's Zhong grammar and DELPH-IN as my parser as it had far more granular rules (high specificity to Chinese specifically rather than just VBN, VBZ, etc) than the statistical parsers I had encountered. – Andrew Alexander Jan 15 '19 at 0:43
1

Ok, so I've got a basic example in code - I'll ultimately write my algorithm to be more efficient and right now it is only written for the one use-case (I haven't generalized it/made it so that you can have multiple daughters under a rule yet), but right now I'm just writing it out for one single rule to show an example of what I ultimately want to do.

If there is a more efficient way to do what I am doing, PLEASE tell me, as I don't know what I don't know on this subject. I am using PyDelphin for the analysis. EDITTED WITH NEWEST VERSION

from delphin.interfaces import ace
zhs = 'zhs.dat'

grammar = zhs

parse_string = '我 看 了 猴子 的 雪白 雪白 的 小小 的 猫'

print("PARSE STRING: " + parse_string)

with ace.AceParser(grammar,
                    cmdargs=['-n', '50']) as parser:
                        tree = parser.interact(parse_string).results()[9].derivation()

# Maybe reverse mapping so that long form POS is key?
pos_mapping = {
    'j': 'adjective',
    'pfv': 'perfective aspect',
}

def redup_post_processing(phrase):
    phrase = phrase.replace('々', phrase)
    phrase = phrase.replace('々', "")
    return phrase

def pos_generate(node):
    pos_level = node.preterminals()[0].entity
    TAG = pos_level.split("_")[1]
    POS = pos_mapping[TAG]
    return POS

def grammar_rule_gen(node, data):
    POS = pos_generate(node)
    phrase = node.terminals()[0].form
    if 'postprocess' in data.keys():
        phrase = data['postprocess'](phrase)
    print(data['text'] + POS + ' ' +  phrase)

functions = {
    'redup-olr': {"function": grammar_rule_gen, "text": "Expressing reduplication with the ", "postprocess": redup_post_processing},
    '了_pfv': {"function": grammar_rule_gen, "text": "Expressing completion with the "},
}

def tree_parse(tree):
    if hasattr(tree, 'daughters'):
        daughters = tree.daughters
        for daughter in daughters:
            if hasattr(daughter, 'entity'):
                if daughter.entity in functions:
                    functions[daughter.entity]['function'](daughter, functions[daughter.entity])
            tree_parse(daughter)

tree_parse(tree)

Now when I run this, it accurately gets me the rule in question that I want - here is my output:

PARSE STRING: 我 看 了 猴子 的 雪白 雪白 的 小小 的 猫
NOTE: parsed 2 / 2 sentences, avg 45873k, time 0.22833s
Expressing completion with the perfective aspect 了
Expressing reduplication with the adjective 雪白雪白
Expressing reduplication with the adjective 小小

And here is the original rule in the grammar (々 is a Japanese character used in Chinese on occasion to represent reduplication - https://en.wiktionary.org/wiki/%E3%80%85):

redup-olr :=
%suffix (* 々)
redup-lex-rule.

And here is the 了 rule:

了_pfv := pfv-marker &
  [ STEM <"了">,
    SYNSEM.LOCAL.CAT.HEAD.CHAR [ FCHAR "了", LENGTH one ],
    SYNSEM.LOCAL.CAT.VAL.COMPS < [ LOCAL.CAT.HEAD.LZG le ] >,
    TRAITS native_token_list ].

Now it would be awesome to just apply these rules as the text was processed - but that would require me writing an extension for ACE or a custom parser (which I guess on some level, I am sorta doing, but nowhere near as complex), and that feels far more complicated than what I am doing here.

Is this the most efficient way to do what I want to do?

This IS a solution to my problem, but I am not sure if it is the most elegant, or computationally cheap.

| improve this answer | |
  • These Python lines of code do not work for me :( – zixuan Jan 26 '19 at 23:47
  • You're using the Zhong grammar and have ACE & PyDelphin installed? – Andrew Alexander Jan 26 '19 at 23:50
  • (sorry ignore my previous comment) Yep. But, it seems like to me that it says "NOTE: parsed 1 / 2 sentences, avg 231451k, time 0.95741s" to me. I've been confused that it only parsed 1 when I run it. – zixuan Jan 26 '19 at 23:52
  • Yeah, I'm not quite sure on the parsing note and why it thinks my string is two sentences - possibly because the tree is split under head-comp: delph-in.github.io/delphin-viz/demo/… – Andrew Alexander Jan 26 '19 at 23:54
  • But right now this is essentially doing what I want it to - I'd need to write rules for each grammar rule I want (which is a bit time intensive, but I only need a couple hundred to start, so it shouldn't be awful). It'd be nice to be able to hook directly into ACE (I guess I could rewrite the Zhong grammar to use my plain english text instead of 'redup-olr', but that seems like a poor idea in case they update the grammar) – Andrew Alexander Jan 26 '19 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.