6

I realize that this is probably technically off-topic as it relates to copyright legalities of corpus construction. However, it is a practically very important issue for corpus linguistics and so I was hoping that I could ask the question here.

In brief, I would like to make a corpus for academic research purposes using publicly accessible news web pages (like BBC news). I am primarily interested in developing statistical models of news topics and how they change over time. I will not distribute the corpus.

The technical aspects of developing this are not a problem. The question I have is whether I am in violation of copyright by making a robot that downloads webpages and statistically analyses their content.

The reason why this is not obviously copyright violation is because the act of downloading and statistically analyzing webpages is essentially what e.g. search engine robots do, and they are not in violation of copyright (see here, for example).

Has anyone been in a similar situation already and have some wisdom to share on the matter?

Improve this question
9
  • Unless you plan to make a lot of money from analyzing this information, how is any copyright holder going to know, and why should they care if they did? The notion of copyright is due for a stiff updating; it's impossible to use any electronic text without copying it from storage to memory, for instance. If you're sitting on a pot of gold and won't share it, they might get troublesome; but that seems fairly unlikely.
    – jlawler
    Oct 17, 2014 at 0:56
  • 1
    I plan to make no money from this and I completely agree that the copyright holder would probably not know and probably not care anyway. However, I am still curious about the legalities, primarily just in case my university ever raises this question.
    – mjandrews
    Oct 17, 2014 at 1:19
  • 1
    IMHO this is perfectly on topic as a practical question concerning corpus creation.
    – robert
    Oct 17, 2014 at 10:31
  • 2
    Copyright specifically limits one's right to copy (publish/distribute). If you aren't publishing or distributing the material, copyright doesn't apply. (That doesn't mean you aren't in violation of some other contractual terms, such as if the BBC only grants access to humans, or something).
    – Flimzy
    Oct 17, 2014 at 12:57
  • 1
    I'm voting to close this question as off-topic because it is about legal interpretation.
    – curiousdannii ♦
    Mar 10, 2015 at 7:35

2 Answers 2

Reset to default
1

I am not in a position to give any legal references, but as far as I know this should be perfectly legal. In fact, many corpora that are not publicly distributed rely on material that the compilers do not have the copyright for. Moreover, Mark Davies, the compiler of the Corpus of Contemporary American English and Corpus of Historical American English thinks it is legal to

(1) provide snippet views for concordance lines

(2) distribute the original texts verbatim with a couple of words every 200 words removed.

He argues that this falls under fair use and provides loads of copyright material in his corpora to other people. What you are planning to do is surely much less of a copyright infringement.

Improve this answer
2
  • 1
    Number (2) sounds dubious to me.
    – curiousdannii ♦
    Oct 18, 2014 at 0:55
  • It is interesting that some corpus compilers have defended their distribution of copyrighted texts in terms of fair use. I am a bit surprised, but it is good to hear. Personally, I am not even planning to distribute the texts, just to download and analyze them.
    – mjandrews
    Oct 19, 2014 at 17:03
0

Questions of legal interpretation can be asked on Law SE. Since this won't get migrated, I'll give the answer I'd give there. First, jurisdiction matters: permission is required under EU law, though non-commercial text mining for research purposes is legal in the UK. In the US, one must do more guessing. Following Authors Guild v. Google, "fair use" is a conceivable defense. (That means, rights holders can sue you and during the trial you can argue that it falls under "fair use"). The problem is that it's hard to predict when "fair use" will work. Here is a summary of the fair use defense. Factors that would rule against a fair use defense are extent (copying a bit vs. copying everything), effect on market (what would the rights-holder have lost?), and purpose (education vs. commercial exploitation). In fact, as far as I know, in the US there has never been a finding of infringement when the use is purely research (no royalties for articles). It would also be important to respect opt-out meta-tags, which expressly deny permission.

There isn't clearly settled law regarding the legality of mechanised copying involved in text-mining, where a bot scrapes text off the interwebs and performs a statistical analysis on that text. Any re-distribution of such copied text (i.e. giving examples of a construction) needs to be quite limited. Any putative escape hatch like "a couple of words every 200 words removed" is total nonsense, as is the "change a few words and you're okay" myth.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.