|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectorg.apache.lucene.index.pruning.PruningPolicy
org.apache.lucene.index.pruning.TermPruningPolicy
org.apache.lucene.index.pruning.RIDFTermPruningPolicy
public class RIDFTermPruningPolicy
Implementation of TermPruningPolicy that uses "residual IDF"
metric to determine the postings of terms to keep/remove, as defined in
http://www.dc.fi.udc.es/~barreiro/publications/blanco_barreiro_ecir2007.pdf.
Residual IDF measures a difference between a collection-wide IDF of a term (which assumes a uniform distribution of occurrences) and the actual observed total number of occurrences of a term in all documents. Positive values indicate that a term is informative (e.g. for rare terms), negative values indicate that a term is not informative (e.g. too popular to offer good selectivity).
This metric produces small values close to [-1, 1], so useful ranges for thresholds under this metrics are somewhere between [0, 1]. The higher the threshold the more informative (and more rare) terms will be retained. For filtering of common words a value of close to or slightly below 0 (e.g. -0.1) should be a good starting point.
| Field Summary |
|---|
| Fields inherited from class org.apache.lucene.index.pruning.TermPruningPolicy |
|---|
fieldFlags, in |
| Fields inherited from class org.apache.lucene.index.pruning.PruningPolicy |
|---|
DEL_ALL, DEL_PAYLOADS, DEL_POSTINGS, DEL_STORED, DEL_VECTOR |
| Constructor Summary | |
|---|---|
RIDFTermPruningPolicy(org.apache.lucene.index.IndexReader in,
Map<String,Integer> fieldFlags,
Map<String,Double> thresholds,
double defThreshold)
|
|
| Method Summary | |
|---|---|
void |
initPositionsTerm(org.apache.lucene.index.TermPositions tp,
org.apache.lucene.index.Term t)
Called when moving TermPositions to a new Term. |
boolean |
pruneAllPositions(org.apache.lucene.index.TermPositions termPositions,
org.apache.lucene.index.Term t)
Prune all postings per term (invoked once per term per doc) |
int |
pruneSomePositions(int docNum,
int[] positions,
org.apache.lucene.index.Term curTerm)
Prune some postings per term (invoked once per term per doc). |
boolean |
pruneTermEnum(org.apache.lucene.index.TermEnum te)
Pruning of all postings for a term (invoked once per term). |
int |
pruneTermVectorTerms(int docNumber,
String field,
String[] terms,
int[] freqs,
org.apache.lucene.index.TermFreqVector v)
Pruning of individual terms in term vectors. |
| Methods inherited from class org.apache.lucene.index.pruning.TermPruningPolicy |
|---|
pruneAllFieldPostings, prunePayload, pruneWholeTermVector |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Constructor Detail |
|---|
public RIDFTermPruningPolicy(org.apache.lucene.index.IndexReader in,
Map<String,Integer> fieldFlags,
Map<String,Double> thresholds,
double defThreshold)
| Method Detail |
|---|
public void initPositionsTerm(org.apache.lucene.index.TermPositions tp,
org.apache.lucene.index.Term t)
throws IOException
TermPruningPolicyTermPositions to a new Term.
initPositionsTerm in class TermPruningPolicytp - input term positionst - current term
IOException
public boolean pruneTermEnum(org.apache.lucene.index.TermEnum te)
throws IOException
TermPruningPolicy
pruneTermEnum in class TermPruningPolicyte - positioned term enum.
IOException
public boolean pruneAllPositions(org.apache.lucene.index.TermPositions termPositions,
org.apache.lucene.index.Term t)
throws IOException
TermPruningPolicy
pruneAllPositions in class TermPruningPolicytermPositions - positioned term positions. Implementations MUST NOT
advance this by calling TermPositions methods that advance either
the position pointer (next, skipTo) or term pointer (seek).t - current term
IOException
public int pruneTermVectorTerms(int docNumber,
String field,
String[] terms,
int[] freqs,
org.apache.lucene.index.TermFreqVector v)
throws IOException
TermPruningPolicy
pruneTermVectorTerms in class TermPruningPolicydocNumber - document numberfield - field nameterms - array of termsfreqs - array of term frequenciesv - the original term frequency vector
IOException
public int pruneSomePositions(int docNum,
int[] positions,
org.apache.lucene.index.Term curTerm)
TermPruningPolicy
pruneSomePositions in class TermPruningPolicydocNum - current document numberpositions - original term positions in the document (and indirectly
term frequency)curTerm - current term
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||