Reading Notes for lecture 6

Task:

1. IIR chapter 8.
2. Karen Sparck Jones, What's the value of TREC: is there a gap to jump or a chasm to bridge?
3. Kalervo Järvelin, Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques ACM Transactions on Information Systems (TOIS) Volume 20 , Issue 4 (October 2002) Pages: 422 – 446

The evaluation of a IR system makes clear that how well the IR system does. Actually, there are altogether 3 field to determine the performance of the IR system:

1. The effectiveness of search result

2. Processing time and space efficiency

3. The satisfaction of users.

The effectiveness of retrieved results are the largest concern, when determing how well a IR system does. Two basic measures are used:

1. Precision: Fraction of retrieved documents that are relevant

2. Recall: Fraction of relevant documents that are retrieved.

There is a tradeoff between P and R: By retrieve more documents, we can increase R, but this might decrease P.

There are several ways to demonstrate the performance:

1.Precision at n documents retrieved:

Show P at fixed point in ranking

2.R-Precision:

P when # of revelant documents retrieved

3.Average Precision

Average P at each relevant document retrieved

4. F-measure

Question:

1. In what level the performance of a IR system can be defined as 'bad'? Is there any standard? for example, when F- measure becomes ***.