I feel like I might be in the wrong exchange, but I do not see a NLP exchange, so I figured this was the best bet. My current task is to take short sentences and try to categorize them based on some taxonomy. I'll try to be as detailed as possible in this question, but if you need any more details, I'll do my best to provide them.
Now there are two main issues with this:
- Dealing with a single sentence, versus having a whole article or document, makes categorizing difficult (as far as I know)
- I do not know what taxonomy I should be trying to use (i.e. a small set of topics, or something larger). I imagine this will be determined based on how accurate I can get this model to work
To give some examples of what I am thinking about:
- "Barack Obama is a Socialist" --> This should be categorized as "Politics", or "US Politics"
- "Global Warming is a Hoax" --> This should be categorized as "global warming", or "global warming debate"
- "Charlie Seen's latest rant about Obama is stupid" --> This should be categorized as "Entertainment", "Hollywood", maybe "Politics".
Research I have done shows that categorizing with short sentences (and not a large article) makes things difficult, and a common way of dealing with it is to utilize wikipedia / dbpedia. What I would like is this:
- Research papers which have tried to categorize sentences, tweets, sm posts, etc into a set of topics
- Common methods for dealing with small amounts of text
- Links to software, apis etc that may help with categorizing (most APIs I have seen work better with documents)
For the record, I know this is a quite a bit vague. I'll try to provide any information needed to help figure out what strategies would work well. Any insights, links and thoughts would be highly appreciated.
