I've recently been tasked with programming a web crawler that searches through news articles for certain keywords--company names, for example--and then tries to determine if the article has a positive or negative slant. The web crawling portion is trivial but I feel that I've bitten off more than I can chew for the textual analysis portion.

My current approach is detailed below. I'm wondering if it's a sensible way to do things, or if there are better methods to try.

The crawler starts off by pulling the page's DOM. It then assumes that the actual "article" is within the paragraph <p> tags, so it reduces the data down to just that text. (This may be refined later on a per-website basis, but for now it works reasonably well). There are three input files: the aforementioned keywords, with words like "Facebook" or "Google", and then two wordlists, one containing positive words and the other containing negative.

The program then crawls through each paragraph of text in search for the keywords. If it finds a keyword, it starts checking every word against both word list. A keyword has a weight for each webpage, starting at 0, and incrementing for positive words found in the paragraph, decrementing for negative words found. So if 5 positive words and 3 negative words are found accompanying a keyword, a weight of +2 is assigned. If the combined weight of all paragraphs is positive (or even above some cut-off) the article is deemed to view it in a positive light, and the opposite for a negative score.

Obviously this approach brings some pretty huge issues. For example, a keyword (always a proper name) can be used once in an article, and the rest of the text may just refer to it as "it". I could make the scoping of the keyword per article (i.e., if the keyword is found anywhere in the page, then any paragraph containing listed words count toward the weight) but that brings it's own problems.

I'm not a linguist, and I realize this problem is probably pretty high up as far as linguistic problems go. I'm not looking for a perfect solution, but I'd like to know if there is anything more I could be doing.

Thanks for any feedback. I'm not sure if this is really appropriate for this stackexchange site, if it's not I'd like to know a better place.

2 Answers 2


Sentiment Classification is a still developing field so I don't feel comfortable saying that there is a right or a wrong way to approach your problem. Your approach would theoretically work but it is a bit simplistic. Consider the following:

First, how will you choose your word list? Will it be manually compiled? How will you make sure it is representative and has wide enough coverage to actually be useful?

Second, how will you choose your coefficients? From your description, I'm assuming all coefficients will be +1 or -1, but surely a word like "excellent" conveys more positivity than a mere "good" and reflecting this in your coefficients would improve accuracy. How will your coefficients reflect this? How do you know they are accurate?

I would suggest you read a few papers on Sentiment Classification (perhaps starting with the Google Scholar link I gave at the beginning of my answer) to get a general understanding of modern techniques.

From what I'm seeing, most solutions use some manually annotated sentiment corpus, like the Multiple Perspective Question Answer (MPQA) Corpus and perform machine learning techniques (usually SVM or MaxEnt) to decide which words and features are strong indicators of sentiment and what their optimal coefficient should be.

There are lots of machine learning libraries to help you with the programming component. A simple Google search should help you find a suitable language for your programming language of choice.

  • So yeah, I've thought about both of those points. The word lists are probably going to be manually chosen. I think I can just peruse a thesaurus to compile a suitable list. I am definitely planning on weighting the scores, though how exactly I do that will probably require some experimentation. I'm glad you brought up an actual field (which I've never heard of despite my searching), and some papers too--all the papers so far on this subject have been very broad without no details on what implementation should look like. Having somewhere to start is going to help me a lot. Thanks!
    – stackptr
    Commented Jul 1, 2013 at 3:44
  • @Corey You saying you want to manually choose your list actually worries me quite a bit. I strongly think you should reconsider the MPQA + machine learning approach I described. I know it sounds complicated but I can almost guarantee it will be easier and more effective than trying to manually compile your adjective list and coefficients. Once you start pointing your program at real world data you might realize that your manually created list isn't as good as you thought.
    – acattle
    Commented Jul 1, 2013 at 15:59
  • I'll definitely look into it, then.
    – stackptr
    Commented Jul 1, 2013 at 20:03

This book by Pang and Lee is a classic reference on the topic. To get a good overview of the topic, look at the book's website which has links to presentation slides.

I see that you want to implement the software on your own, and you want to (re-)invent algorithms too. The reasons behind these is not entirely clear. If you are writing a research paper or a thesis on the topic, you would need to do both. Otherwise, you can (1) look for free software for sentiment analysis; if you can't find any, or the licenses are unsuitable for your use, (2) learn techniques from published papers that demonstrate good results for the kind of classification you wish to do. There are a few workshops on this topic, and the website keeps changing from year to year, based on who is hosting it. In general, you could look for papers at "Workshop on Computational Approaches to Subjectivity and Sentiment Analysis" (WASSA-13, for example) or at "Practice and Theory of Opinion Mining and Sentiment Analysis" (PATHOS-2013, for example).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.