1

I am not sure if this is the right place to post or if I should be posting CompSci so please feel free to redirect me to the most appropriate StackExchange.

I would like to determine interests or hobbies that people have given two sources of data from their Twitter data. Those two pieces of information are their Bio, which is a user supplied description of themselves and their tweets (both a maximum of 140 characters).

I don't have background in NLP but intuitively I have been thinking of three contexts for determining interests given here with decreasing confidence:

  1. Using explicit word phrases that express interest: interested in, passionate about, etc.
  2. A positive sentiment expressed about the subject.
  3. Neutral or even negative sentiment for a topic (for example, somebody may be interested in their football team but dislike their current manager).

The result I am trying to achieve is to automatically label a user with a collection of interests such as:

  • Finance
  • Programming

And/or more specific examples like:

  • FOREX Trading
  • Android Development

I have tried using LDA, specifically the topicsmodels package in R, using an individuals tweets as the corpus but I am not so sure about the applicability of the results. LDA gives a collection of terms associated with the topics it found. This requires some manual interpretation to distinguish what the interest is.

For example you might get some results from LDA like the following:

  • [Twitter, Java, time, work, day]
  • [code, IPhone, mobile, look, job]
  • [football, fantasy, team, pick, score]

I can generally pick out the topics people are interested in here but its just the key word or topic identifier I am interested not the other contextual terms like team, pick or score in the last example.

Perhaps part of speech tagging would be useful, maybe I should filter by noun, proper noun, proper name to get the core interest?

Also I would need to handle ambiguous topics such as Java, which could be a programming language, coffee variety or the island itself.

Any suggestions, references, links or pointers would be appreciated. I am comfortable with R, Java and Python if there are libraries which could help achieve this.

Improve this question
3
  • I agree that you are probably much more likely to get useful answers at SE CompSci.
    – robert
    Nov 22, 2013 at 14:59
  • This is a subfield of natural language processing called sentiment analysis, so technically it's on topic. But it's also on-topic on at least one other Stack Exchange site. So if you don't find any useful answers coming here then I would also recommend asking it on one of the other applicable sites.
    – hippietrail
    Nov 23, 2013 at 15:43
  • I realize this post is old as balls, but I wonder if an NLP type analysis could be combined with a social network analysis. People tend to have similar interests as their followers/those they're following.
    – Russell Richie
    Jan 18, 2016 at 23:48

1 Answer 1

Reset to default
1

At my previous company we used the Stanford sentiment algos - it was a mixed blessing, took a fair amount of tweaking and calibration: http://nlp.stanford.edu/sentiment/index.html

For Twitter, it was surprisingly reliable as the short discourse limits ambiguity.

Other than that, several machine learning algos are there for you to train. Burak Kanber has a pretty good blog around it - though he used JS for a naive Bayes classifier: http://burakkanber.com/blog/machine-learning-naive-bayes-1/

Limitations of machine based solutions: sarcasm, ambiguity, cynicism, metaphors and similar mythical beast are still beyond its capabilities.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.