I am currently writing my master thesis about extracting "ideas" for innovation from text stored digitally. Thus the project is a combination of "Marketing", "Datamining/statistics" and "Linguistics". I am a marketing student and the last english class I had was 6-7 years ago.
I have come so far that I need to understand what "semantics" (having updated this I am not so sure anymore, but I leave I for now) are related to discussions about ideas. To give a couple of examples of what I mean one could write the following idea in a forum:
I came up with an idea of how to build a car that is fueled by water.
Yesterday I came up with a cool idea about how to create a cure for cancer.
My point of showing these two example is that if I was to run some statistical analysis on these two examples, the nouns "car", "car", "water", "cure" and "cancer" does not really contribute to whether a sentence contains what I am looking for. So I might as well throw these out. However the phrase "came up with" does provide a lot of information as this is one way to express that one have an idea.
One could imagine several variations of these examples but my point is that basically one should be able to remove nouns as they should not make a difference in theory. Even the word "idea" which is also a noun one should be able to remove, but maybe one need to distinguis between nouns that is often connected to discussions about ideas and nouns in general. For example the nouns "idea", "innovation", "design" and "model" might be nouns that in some degree might show up when people discuss ideas.
To give an example of why I started searching this way I have already build a model (via SVM) based on n-grams. When one create the model one obtains the weights of a given term or n-gram that contributes. When looking through that list, n-grams like
"that sounds cool",
"i have" and/or
"i had an" tend to weight highly, getting me to think that dividing the text into ngrams/chunks is a part of how one should model this.
As already mentioned one might have to distinguish bewteen nouns which are likely to show up when people debate ideas but unlikely to show up when people debate ideas within a given domain. I will give a couple of examples below:
It would be a great idea to invent chocolate that make you loose weight
I am working with innovation and I have just come up with an idea of how to produce chocolate that makes you loose weight
So the following phrases/n-grams are likely to have positive weights:
"it would be",
"have just come up with",
"just come up with",
"come up with"
One could also imagine that the verb
"invent" would also have positive weights.
Finally the nouns
"innovation" are also likely to have positive weights, but the term
chocolate is not as it only tells us what domain of ideas we are dealing with.
The case is that I was handed over 300 classified posts from a userforum where three judges had classified the posts. The posts was classified as containing an idea(1) or not (0) and if at least two of the judges agreed that there was an idea in the text I would use it as a positive case. This gives me 12 positive cases and 288 negative cases, which is of course a very bad training set far to small and unbalanced to do anything proper with. Then the thought was to use this small bad training sample to create a preliminary filter that can be used to extract around 3000 posts from the original forum with a higher likelihood of containing an idea. What I am doing now is that I am trying to optimize the preliminary filter to get a good training set for the final model.
I am writing this here because I do not have a clue about how to attack this problem. I know how to setup a computer to do what I want it to, but I dont know how to work with language in a structured way. Therefore I hope that someone in here could throw me a couple of keywords and maybe even sources that I can look at myself in order to figure out how to structure my problem better. I would also be nice just to have someone to debate this with as I am totally on my own.