I've recently been tasked with programming a web crawler that searches through news articles for certain keywords--company names, for example--and then tries to determine if the article has a positive or negative slant. The web crawling portion is trivial but I feel that I've bitten off more than I can chew for the textual analysis portion.
My current approach is detailed below. I'm wondering if it's a sensible way to do things, or if there are better methods to try.
The crawler starts off by pulling the page's DOM. It then assumes that the actual "article" is within the paragraph <p>
tags, so it reduces the data down to just that text. (This may be refined later on a per-website basis, but for now it works reasonably well). There are three input files: the aforementioned keywords, with words like "Facebook" or "Google", and then two wordlists, one containing positive words and the other containing negative.
The program then crawls through each paragraph of text in search for the keywords. If it finds a keyword, it starts checking every word against both word list. A keyword has a weight for each webpage, starting at 0, and incrementing for positive words found in the paragraph, decrementing for negative words found. So if 5 positive words and 3 negative words are found accompanying a keyword, a weight of +2 is assigned. If the combined weight of all paragraphs is positive (or even above some cut-off) the article is deemed to view it in a positive light, and the opposite for a negative score.
Obviously this approach brings some pretty huge issues. For example, a keyword (always a proper name) can be used once in an article, and the rest of the text may just refer to it as "it". I could make the scoping of the keyword per article (i.e., if the keyword is found anywhere in the page, then any paragraph containing listed words count toward the weight) but that brings it's own problems.
I'm not a linguist, and I realize this problem is probably pretty high up as far as linguistic problems go. I'm not looking for a perfect solution, but I'd like to know if there is anything more I could be doing.
Thanks for any feedback. I'm not sure if this is really appropriate for this stackexchange site, if it's not I'd like to know a better place.