I am currently writing my master thesis about extracting "ideas" for innovation from text stored digitally. Thus the project is a combination of "Marketing", "Datamining/statistics" and "Linguistics". I am a marketing student and the last english class I had was 6-7 years ago.

I have come so far that I need to understand what "semantics" (having updated this I am not so sure anymore, but I leave I for now) are related to discussions about ideas. To give a couple of examples of what I mean one could write the following idea in a forum:

I came up with an idea of how to build a car that is fueled by water.

Another example:

Yesterday I came up with a cool idea about how to create a cure for cancer.

My point of showing these two example is that if I was to run some statistical analysis on these two examples, the nouns "car", "car", "water", "cure" and "cancer" does not really contribute to whether a sentence contains what I am looking for. So I might as well throw these out. However the phrase "came up with" does provide a lot of information as this is one way to express that one have an idea.

One could imagine several variations of these examples but my point is that basically one should be able to remove nouns as they should not make a difference in theory. Even the word "idea" which is also a noun one should be able to remove, but maybe one need to distinguis between nouns that is often connected to discussions about ideas and nouns in general. For example the nouns "idea", "innovation", "design" and "model" might be nouns that in some degree might show up when people discuss ideas.

To give an example of why I started searching this way I have already build a model (via SVM) based on n-grams. When one create the model one obtains the weights of a given term or n-gram that contributes. When looking through that list, n-grams like "that would", "that sounds cool", "i have" and/or "i had an" tend to weight highly, getting me to think that dividing the text into ngrams/chunks is a part of how one should model this.

As already mentioned one might have to distinguish bewteen nouns which are likely to show up when people debate ideas but unlikely to show up when people debate ideas within a given domain. I will give a couple of examples below:

It would be a great idea to invent chocolate that make you loose weight


I am working with innovation and I have just come up with an idea of how to produce chocolate that makes you loose weight

So the following phrases/n-grams are likely to have positive weights:

"it would be", "have just come up with", "just come up with", "come up with"

One could also imagine that the verb "invent" would also have positive weights.

Finally the nouns "idea" and "innovation" are also likely to have positive weights, but the term chocolate is not as it only tells us what domain of ideas we are dealing with.

The case is that I was handed over 300 classified posts from a userforum where three judges had classified the posts. The posts was classified as containing an idea(1) or not (0) and if at least two of the judges agreed that there was an idea in the text I would use it as a positive case. This gives me 12 positive cases and 288 negative cases, which is of course a very bad training set far to small and unbalanced to do anything proper with. Then the thought was to use this small bad training sample to create a preliminary filter that can be used to extract around 3000 posts from the original forum with a higher likelihood of containing an idea. What I am doing now is that I am trying to optimize the preliminary filter to get a good training set for the final model.

I am writing this here because I do not have a clue about how to attack this problem. I know how to setup a computer to do what I want it to, but I dont know how to work with language in a structured way. Therefore I hope that someone in here could throw me a couple of keywords and maybe even sources that I can look at myself in order to figure out how to structure my problem better. I would also be nice just to have someone to debate this with as I am totally on my own.

  • 1
    SVM is definitely an excellent way to go about it. But, if you have already made it, why are you trying to look beyond it? What, about its performance, are you dissatisfied with?
    – prash
    Commented Dec 14, 2012 at 10:27
  • Hi prash. I have added a section to my original question that should answer your question. In short I am trying to build a model based on a small bad training set that can help me extract enough positive cases to get a more balanced dataset. Commented Dec 14, 2012 at 11:21
  • If I understand correctly, you are looking for a way to increase your training set. Right? Even the chunkers I mentioned earlier might be overkill for the task. Have you tried regular expressions? If you don't have much luck with that, you might want to look at one of the chunkers that did well with verb phrases.
    – prash
    Commented Dec 14, 2012 at 15:44
  • Well I know regular expressions but I have not thought about regular expressions as the solution to increasing my training set. I basically only use it when i have some string that I want to extract something from. Please elaborate... I have around 440.000 cases stores as text files and with the model I have so far, I get reasonable results when I use the model for classifying the rest of the posts. I get around 7200 cases with a confidence above 0.50 of containing an idea. Commented Dec 14, 2012 at 20:27
  • Assuming you have one sentence per line, grep "((it would be)|(come up with)|(came up with))" file-with-sentences.txt would give you a list of all the sentences that match your criteria. This probably seems obvious to you, but from your last comment, I get the impression that this is what you are asking for. Once you have a shorter list of sentences, you can prune it manually.
    – prash
    Commented Dec 14, 2012 at 21:08

2 Answers 2


Your question sounds simple, but it unveils an entire area of human activity, a Natural Language Processing.

The problem is that natural languages convey the content in a linked manner. It means that a phrase "build a car fueled by water" can't stand alone. Instead, it is bound to various objects, like this: I (came up with (an idea of (how (to build a car (that is fueled by water))))). An actor, "I" is linked to an idea ("a car that is...") through a hierarchy of relations, or links. Hence, your data mining system has to follow the entire sequence, otherwise it would be quickly confused with phrases like this:

I have no clue how to build a car that is fueled by water.

As you see, the actor and the idea are essentially the same, but the linkage between them indicates the whole opposite message.

Yet another example:

I don't think that inventing a car that is fueled by water is a total nonsense.

Although a naive algorithm will find negative words ("nonsense", "I don't think"), the entire message conveys a positive idea.

Finally, you will end up with a semantic graph, like this:

Semantic graph

(image produced by the interface that provides online access to the LinGO English Resource Grammar).

You may have notice that most common phrases have various ways to be parsed into a semantic graph. You have to use some algorithms figuring out the most probable variant.

Please don't be discouraged by the complexity; it's not that terrible. :) For most practical tasks, you don't have to get into the full depth of analysis. However, knowing theoretical key points will let you focus your attention on practical implementation.

Also, I would suggest attending some courses that can provide you with some basis. For instance, Coursera has free course of NLP, and they also have a series of videos that may be useful.

  • 2
    See also Framenet.
    – jlawler
    Commented Dec 12, 2012 at 20:40
  • Thanks for your answer. I am actually already signed up for the Coursera NLP course. Looking forward to it. The problems you are talking about is exactly what I think I need to figure out how to distinguish between. For a while i have been trying to build a model with only "uni-grams" which proved to be no good at all. I them shifted into using "n-grams" which hopefully will giver other results. I am using Rapid Miner for the modeling part but can also use R if necessary but i don not think that R has enough NLP opportunities at this point. Commented Dec 12, 2012 at 21:04
  • 1
    @bytebuster: The graph you have attached shows only syntax. With semantics, you'd have a graph that also shows the MRS of the sentence.
    – prash
    Commented Dec 13, 2012 at 20:23
  • @prash Absolutely. Please consider expanding your comment into an full answer as it would provide a great information for the asker. Commented Dec 13, 2012 at 21:02
  • @bytebuster: MRS and I haven't been formally introduced to each other. We haven't even worked together. To add to that, I believe deep analysis is the wrong way to go -- because Kasper intends to ignore much of the parse information. HPSG and Framenet are fine ideas, but HPSG would be overkill and Framenet (AFAIK) does not have seem to have tools that utilize its resources. The tool I have seen that comes closest is Boxer, that can emit Framenet roles.
    – prash
    Commented Dec 14, 2012 at 0:07

From the description of your task, I get the impression that you do not really need semantics of any kind — you intend to discard many segments of a sentence. I believe that you'd be better off looking at the chunkers listed here.

Chunkers have differing performance levels for noun phrases and verb phrases. You could start with the ones that perform better with verb phrases. You could also try to play with NLTK's chunker. You'd have to define your own patterns for NLTK's chunker.

I haven't worked with any of these tools, myself. So I can't tell you which ones require you to write software. I understand that you are a student of marketing. I don't know how comfortable you'd be with writing code. But if you have to write code, I think writing Python (for NLTK) should be less of a hair-pulling experience for you, than writing Java for one of the Java-based tools could be.

  • Thar definately looks useful. Thanks a lot. And yes I am a student of marketing, but a special case as i have chosen to specialize within business intelligence and data mining/textmining. I am using R for my project and have become fairly familiar with that. I know that python is "the" language for NLP but R is doing the job fine for me. Commented Dec 14, 2012 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.