I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I was hoping that I could ask the question here.
In brief, I would like to make a corpus for academic research purposes using publicly accessible news web pages (like BBC news). I am primarily interested in developing statistical models of news topics and how they change over time. I will not distribute the corpus.
The technical aspects of developing this are not a problem. The question I have is whether I am in violation of copyright by making a robot that downloads webpages and statistically analyses their content.
The reason why this is not obviously copyright violation is because the act of downloading and statistically analyzing webpages is essentially what e.g. search engine robots do, and they are not in violation of copyright (see here, for example).
Has anyone been in a similar situation already and have some wisdom to share on the matter?