I'm working on a NLP project, where we analyze large text samples (think a novel), and produce some metrics that help us answer interesting questions about the text. One of these metrics is average number of independent clauses per compound sentence.

The way I determine that a sentence is compound, as opposed to simple or complex (including complex/compound), is by first using the Stanford Parser to obtain the syntax tree, and then doing a variation of the algorithm suggested here. My algorithm is a bit simpler because I wanted to see some numbers over a variety of texts sooner, before deciding if they were interesting enough to invest time in a more complicated approach.

What I found, across genre and register, was that the average number of independent clauses per compound sentence tended to be around 2. It struck me as weird, because I was expecting more variation, but I don't know if it's actually weird. I tried to find information online about this, but that search didn't produce anything helpful.

Is there any research on the prevalence of different sentence types over, say, a text corpus; furthermore, is there any research on sentence complexity? I want to understand if my algorithm is simply buggy, or if it's reflecting an actual trend that others have already seen as well.

  • 1
    Have you tested your algorithm on a small subset of sentences, so you can review the results by hand? Have you tried feeding in sentences that you believe should have more than 2 independent causes, to see the results? Finally, rather than dealing in generalities, if you'd like advice on your code, please post it. – Adam_G Aug 17 '15 at 11:55
  • It should be over 2, not under, if the Ss have already been declared compound. But it shouldn't be very far over 2. By far the majority of compound Ss are 2-clause A and/but/or B Ss. 3-clause Ss occur, for instance, but not often, especially in speech. And @Adam_G has it right about reviewing the input and output by hand to start with. – jlawler Aug 17 '15 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.