Muddiest points for Lecture 6

Question:

In what level the performance of a IR system can be defined as 'bad'? Is there any standard? for example, when F- measure becomes ***, the system is considered as a bad IR system?

Reading Notes for Lecture 7

Task:
1. IIR Chapter 9.
2. Xu, J. and Croft, W. B. 2000. Improving the effectiveness of information retrieval with local context analysis. ACM Trans. Inf. Syst. 18, 1 (Jan. 2000), 79-112.
3. Wang, X., Fang, H., and Zhai, C. 2008. A study of methods for negative relevance feedback. In Proceedings of the 31st Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Singapore, Singapore, July 20 - 24, 2008). SIGIR '08. ACM, New York, NY, 219-226.
4. Donna Harman, Relevance feedback revisited. Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval. Pages: 1 - 10. Copenhagen, Denmark. 1992.

The idea of relevance feedback is to involve the user in the retrieval process so as to improve the final result set. In particular, the user gives feedback on the relevance of documents in an initial set of results. The basic procedure is:

• The user issues a (short, simple) query.
• The system returns an initial set of retrieval results.
• The user marks some returned documents as relevant or nonrelevant.
• The system computes a better representation of the information need based on the user feedback.
• The system displays a revised set of retrieval results.

Relevance feedback can go through one or more iterations of this sort. The process exploits the idea that it may be difficult to formulate a good query when you don't know the collection well, but it is easy to judge particular documents, and so it makes sense to engage in iterative query refinement of this sort. In such a scenario, relevance feedback can also be effective in tracking a user's evolving information need: seeing some documents may lead users to refine their understanding of the information they are seeking.
The idea of relevance feedback ( ) is to involve the user in the retrieval process so as to improve the final result set. In particular, the user gives feedback on the relevance of documents in an initial set of results. The basic procedure is:

• The user issues a (short, simple) query.
• The system returns an initial set of retrieval results.
• The user marks some returned documents as relevant or nonrelevant.
• The system computes a better representation of the information need based on the user feedback.
• The system displays a revised set of retrieval results.

Relevance feedback can go through one or more iterations of this sort. The process exploits the idea that it may be difficult to formulate a good query when you don't know the collection well, but it is easy to judge particular documents, and so it makes sense to engage in iterative query refinement of this sort. In such a scenario, relevance feedback can also be effective in tracking a user's evolving information need: seeing some documents may lead users to refine their understanding of the information they are seeking.
The idea of relevance feedback ( ) is to involve the user in the retrieval process so as to improve the final result set. In particular, the user gives feedback on the relevance of documents in an initial set of results. The basic procedure is:

• The user issues a (short, simple) query.
• The system returns an initial set of retrieval results.
• The user marks some returned documents as relevant or nonrelevant.
• The system computes a better representation of the information need based on the user feedback.
• The system displays a revised set of retrieval results.

Relevance feedback can go through one or more iterations of this sort. The process exploits the idea that it may be difficult to formulate a good query when you don't know the collection well, but it is easy to judge particular documents, and so it makes sense to engage in iterative query refinement of this sort. In such a scenario, relevance feedback can also be effective in tracking a user's evolving information need: seeing some documents may lead users to refine their understanding of the information they are seeking.
Pseudo relevance feedback,also known as blind relevance feedback , provides a method for automatic local analysis. It automates the manual part of relevance feedback, so that the user gets improved retrieval performance without an extended interaction.
Query expansion(QE) is the process of reformulating a seed query to improve retrieval performance in information retrieval operations. In the context of web search engines, query expansion involves evaluating a user's input (what words were typed into the search query area, and sometimes other types of data) and expanding the search query to match additional documents. Query expansion involves techniques such as:

• Finding synonyms of words, and searching for the synonyms as well
• Finding all the various morphological forms of words by stemming each word in the search query
• Fixing spelling errors and automatically searching for the corrected form or suggesting it in the results
• Re-weighting the terms in the original query

Query expansion is a methodology studied in the field of computer science, particularly within the realm of natural language processing and information retrieval.
Q1:
I am curious about the realization of finding synonyms, if it is realized by creating synonyms list for a word? or by other ways?

Muddiest Point for Lecture 5

The Probabilistic ranking theroy is quite confusing, but I think the implementation should be much easier than I previously thought, because all the probability work is done by program.
My question is Can the probablity of relevance be biased if there is relevance between two terms, for example, synonym?

Reading Notes for lecture 6

Task:

1. IIR chapter 8.
2. Karen Sparck Jones, What's the value of TREC: is there a gap to jump or a chasm to bridge?
3. Kalervo Järvelin, Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques ACM Transactions on Information Systems (TOIS) Volume 20 , Issue 4 (October 2002) Pages: 422 – 446

The evaluation of a IR system makes clear that how well the IR system does. Actually, there are altogether 3 field to determine the performance of the IR system:

1. The effectiveness of search result

2. Processing time and space efficiency

3. The satisfaction of users.

The effectiveness of retrieved results are the largest concern, when determing how well a IR system does. Two basic measures are used:

1. Precision: Fraction of retrieved documents that are relevant

2. Recall: Fraction of relevant documents that are retrieved.

There is a tradeoff between P and R: By retrieve more documents, we can increase R, but this might decrease P.

There are several ways to demonstrate the performance:

1.Precision at n documents retrieved:

Show P at fixed point in ranking

2.R-Precision:

P when # of revelant documents retrieved

3.Average Precision

Average P at each relevant document retrieved

4. F-measure

Question:

1. In what level the performance of a IR system can be defined as 'bad'? Is there any standard? for example, when F- measure becomes ***.

Muddiest Point for lecture 4

The lecture about boolean and vector space model is very usful, especially the vector space part. However, 1.I'm still confused about the SMART term weighting, what's the advantage?
2.How can we determine the start point and end point of vector?

Reading Notes for Lecture5

Task:

1.IIR Chapter 11 and Chapter 12

If we have some known relevant and nonrelevant documents, then we can straightforwardly start to estimate the probability of a term t appearing in a relevant document, and that this could be the basis of a classifier that decides whether documents are relevant or not. Chapter 11 introduced some probablistic-based IR model, such as Binary Inde-Model and some more advance one like Bayesian Network Model. I'm wondering that there is the fact that some terms might be relevent between themselves, if the probablity of relevance might be biased if calculated in the way that ignoring the relevance between terms?
The language modeling approach to IR, in Chapter 12, directly models the idea: a document is a good match to a query if the document model is likely to generate the query, which will in turn happen if the document contains the query words often. Such language model includes Query Likihood Model. I think the language modeling is still based on probability, but adding some probabilistic relevance between the sequence of  query and document.