1

BM25 uses this nonlinear transformation to scale the term frequency:

enter image description here

where x is a term frequency. Why is k+1 used in numerator instead of just k?

Wouldn't formula below work equally well?

enter image description here

It is simpler to calculate then the first formula and the resulting scaling is similar as shown on this graph:

enter image description here

where the first formula is drawn in red and the second in blue for k in {1, 2, 3, 4}.

EDIT

Is it because the first formula reduces to 1 (no scaling) for k = 0 and the second one can't be reduced to 1? Turning the scaling on and off could be useful in experimenting with different values of k, especially in the case where other formulas with their own parameters are involved.

  • A comment to explain why it is unclear probably wouldn't hurt. – Franck Dernoncourt Oct 31 '15 at 0:45
  • @FranckDernoncourt I'm not sure what else I can add to my question. I would expect two kinds of answers: (1) xk/(x+k) can't be used because .... or (2) both formulas would work, but x(k+1)/(x+k) was chosen because ... – Paul Jurczak Oct 31 '15 at 4:50