Do I have copyright issues when making a corpus from the web?

Question

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I was hoping that I could ask the question here.

In brief, I would like to make a corpus for academic research purposes using publicly accessible news web pages (like BBC news). I am primarily interested in developing statistical models of news topics and how they change over time. I will not distribute the corpus.

The technical aspects of developing this are not a problem. The question I have is whether I am in violation of copyright by making a robot that downloads webpages and statistically analyses their content.

The reason why this is not obviously copyright violation is because the act of downloading and statistically analyzing webpages is essentially what e.g. search engine robots do, and they are not in violation of copyright (see here, for example).

Has anyone been in a similar situation already and have some wisdom to share on the matter?

Answer 1

I am not in a position to give any legal references, but as far as I know this should be perfectly legal. In fact, many corpora that are not publicly distributed rely on material that the compilers do not have the copyright for. Moreover, Mark Davies, the compiler of the Corpus of Contemporary American English and Corpus of Historical American English thinks it is legal to

(1) provide snippet views for concordance lines

(2) distribute the original texts verbatim with a couple of words every 200 words removed.

He argues that this falls under fair use and provides loads of copyright material in his corpora to other people. What you are planning to do is surely much less of a copyright infringement.

Answer 2

Questions of legal interpretation can be asked on Law SE. Since this won't get migrated, I'll give the answer I'd give there. First, jurisdiction matters: permission is required under EU law, though non-commercial text mining for research purposes is legal in the UK. In the US, one must do more guessing. Following Authors Guild v. Google, "fair use" is a conceivable defense. (That means, rights holders can sue you and during the trial you can argue that it falls under "fair use"). The problem is that it's hard to predict when "fair use" will work. Here is a summary of the fair use defense. Factors that would rule against a fair use defense are extent (copying a bit vs. copying everything), effect on market (what would the rights-holder have lost?), and purpose (education vs. commercial exploitation). In fact, as far as I know, in the US there has never been a finding of infringement when the use is purely research (no royalties for articles). It would also be important to respect opt-out meta-tags, which expressly deny permission.

There isn't clearly settled law regarding the legality of mechanised copying involved in text-mining, where a bot scrapes text off the interwebs and performs a statistical analysis on that text. Any re-distribution of such copied text (i.e. giving examples of a construction) needs to be quite limited. Any putative escape hatch like "a couple of words every 200 words removed" is total nonsense, as is the "change a few words and you're okay" myth.