1

I am doing a text classification task. My text is questions and answers in a community question answering website. I want to extract tags from the title , existing tags, and BODY of the questions and then use supervised learning to learn the tags for each class/category.

There is lot of noise in the BODY (punctuation marks, fully qualified names of Java exceptions, logs etc). How can I deal with such noise to extract good tags from them. Will some thing like Lucene or OpenNLP or any other library serve my purpose. And what techniques should I use.

How to capture intention of the user typing the question in NLP. Like for example, Robert understood my question and decided that its programming one and not linguistics. That is the problem I am trying to solve. How to automate that natural language understanding of text in community question answering sites?

Improve this question
7
  • 2
    This question appears to be off-topic because it is about programming. Will probably find better answers on StackOverflow. – robert Jun 13 '14 at 8:43
  • 1
    I think it's a fair question in the computational linguistics category. Having said that, the question could do with some refinement. Can you (user4654) provide a couple of examples so we can see what kind of text you're considering noise? – dmh Jun 14 '14 at 0:14
  • @robert can you justify your statement ????????? – Vineel Jun 14 '14 at 6:06
  • 2
    You have a very specific, computational problem, and you're even asking what library you could use ("Lucene or OpenNLP or any other library") - it's a programming problem, not a linguistics question (Which could be something like 'What is the typical structure of a question?'). Also, knowledge in NLP is thin on the ground here in SE Linguistics (I'd be happy to be proven wrong by insightful answers to the Q :) – robert Jun 14 '14 at 9:48
  • 2
    The only linguists that could help with this are going to be very active and knowledgeable in CL and NLP. It appears there's not so much here. There are lots of text classification questions on Stack Overflow and a much bigger audience. – hippietrail Jun 15 '14 at 1:09
2

Without seeing your corpus, it is difficult for me to recommend a method you could use. The issue you brought up is not NLP, per se, because it involves preparing a corpus for NLP. However the same set of tools can be used for filtering out "noise".

  1. Normally, source code and console output are formatted differently, with a "<pre>" or with custom CSS. You could write regular expressions or use HTML parsers in your favourite language to filter them out.
  2. If this task is going to be part of a larger tool-chain, GATE is a decent way to do things. Its Jape plugin is what you're likely to use.
Improve this answer
1
  • actually I am trying to build a classifier that classifies questions into domains that they should fall into. For example, reading my question above Robert pointed out that its a programming question and not a linguistics one. I want to automate this process. I have Q&A website questions and their labels too. This is my training data, based on this training data, I want to now classify new questions i.e, assign their labels by identifying the intention of the user. Can you please advise me, what should I do apart from the keyword extraction. – Vineel Jun 14 '14 at 22:36
1

Perl's Lingua::EN::Tagger can extract noun phrases and guess at unknown parts-of-speech, although I'm not sure if that's what you meant. It could probably handle things like exception names.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.