Reading Notes for week 2&3(Jan 24th)

Task:
1.IIR sections 1.2, chapters 2 and 3.

The reading materials introduced some basic topics on inverted index, including the steps in building an inverted index, several approaches used in processing documents, such as how to do tokenization, and how to process querys and deal with typographical errors in the query. The information provided by IIR is detial enough for me to get an overall understanding of the logics in building an inverted index.

The major steps and related document processing and query processing techniques during each steps are:
1. Collect the documents to be indexed
{ Obtaining char sequence in a doc(decode); Choosing a document unit(precision/recall tradeoff)}
2. Tokenization and linguistic preprocessing.
{char-->token(For english: use whitespace, throw punctuations. However, there are such symbols as apostrophe to consider. Actually, how to tokenize the char depends on what kind of language is used)-->type(class of all tokens containing the same character sequence)-->term(Dropping common terms,such as of, a. Normalization(equivalence classing of terms).Stemming and lemmatization(words with similar meanings)list intersection(skip pointers).positional posting and phrase query(biwords, position index, combination schema)}3.Index documents that each term occurs in.

Though the description is quite detail, it only introduce the techniques individually. How the techniques are combined in practice? And all the information are about boolean query, then How about the Free text query?what kind of processing method applied?