|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectorg.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.query.QueryAutoStopWordAnalyzer
public final class QueryAutoStopWordAnalyzer
An Analyzer used primarily at query time to wrap another analyzer and provide a layer of protection
which prevents very common words from being passed into queries.
For very large indexes the cost of reading TermDocs for a very common word can be high. This analyzer was created after experience with a 38 million doc index which had a term in around 50% of docs and was causing TermQueries for this term to take 2 seconds.
Use the various "addStopWords" methods in this class to automate the identification and addition of stop words found in an already existing index.
| Field Summary | |
|---|---|
static float |
defaultMaxDocFreqPercent
|
| Constructor Summary | |
|---|---|
QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate)
Deprecated. Stopwords should be calculated at instantiation using one of the other constructors |
|
QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader)
Creates a new QueryAutoStopWordAnalyzer with stopwords calculated for all indexed fields from terms with a document frequency percentage greater than defaultMaxDocFreqPercent |
|
QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
Collection<String> fields,
float maxPercentDocs)
Creates a new QueryAutoStopWordAnalyzer with stopwords calculated for the given selection of fields from terms with a document frequency percentage greater than the given maxPercentDocs |
|
QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
Collection<String> fields,
int maxDocFreq)
Creates a new QueryAutoStopWordAnalyzer with stopwords calculated for the given selection of fields from terms with a document frequency greater than the given maxDocFreq |
|
QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
float maxPercentDocs)
Creates a new QueryAutoStopWordAnalyzer with stopwords calculated for all indexed fields from terms with a document frequency percentage greater than the given maxPercentDocs |
|
QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
int maxDocFreq)
Creates a new QueryAutoStopWordAnalyzer with stopwords calculated for all indexed fields from terms with a document frequency greater than the given maxDocFreq |
|
| Method Summary | |
|---|---|
int |
addStopWords(IndexReader reader)
Deprecated. Stopwords should be calculated at instantiation using QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader) |
int |
addStopWords(IndexReader reader,
float maxPercentDocs)
Deprecated. Stowords should be calculated at instantiation using QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, float) |
int |
addStopWords(IndexReader reader,
int maxDocFreq)
Deprecated. Stopwords should be calculated at instantiation using QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, int) |
int |
addStopWords(IndexReader reader,
String fieldName,
float maxPercentDocs)
Deprecated. Stowords should be calculated at instantiation using QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, Collection, float) |
int |
addStopWords(IndexReader reader,
String fieldName,
int maxDocFreq)
Deprecated. Stowords should be calculated at instantiation using QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, Collection, int) |
Term[] |
getStopWords()
Provides information on which stop words have been identified for all fields |
String[] |
getStopWords(String fieldName)
Provides information on which stop words have been identified for a field |
TokenStream |
reusableTokenStream(String fieldName,
Reader reader)
Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. |
TokenStream |
tokenStream(String fieldName,
Reader reader)
Creates a TokenStream which tokenizes all the text in the provided Reader. |
| Methods inherited from class org.apache.lucene.analysis.Analyzer |
|---|
close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Field Detail |
|---|
public static final float defaultMaxDocFreqPercent
| Constructor Detail |
|---|
@Deprecated
public QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate)
delegate - The choice of Analyzer that is used to produce the token stream which needs filtering
public QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader)
throws IOException
defaultMaxDocFreqPercent
matchVersion - Version to be used in StopFilterdelegate - Analyzer whose TokenStream will be filteredindexReader - IndexReader to identify the stopwords from
IOException - Can be thrown while reading from the IndexReader
public QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
int maxDocFreq)
throws IOException
matchVersion - Version to be used in StopFilterdelegate - Analyzer whose TokenStream will be filteredindexReader - IndexReader to identify the stopwords frommaxDocFreq - Document frequency terms should be above in order to be stopwords
IOException - Can be thrown while reading from the IndexReader
public QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
float maxPercentDocs)
throws IOException
matchVersion - Version to be used in StopFilterdelegate - Analyzer whose TokenStream will be filteredindexReader - IndexReader to identify the stopwords frommaxPercentDocs - The maximum percentage (between 0.0 and 1.0) of index documents which
contain a term, after which the word is considered to be a stop word
IOException - Can be thrown while reading from the IndexReader
public QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
Collection<String> fields,
float maxPercentDocs)
throws IOException
matchVersion - Version to be used in StopFilterdelegate - Analyzer whose TokenStream will be filteredindexReader - IndexReader to identify the stopwords fromfields - Selection of fields to calculate stopwords formaxPercentDocs - The maximum percentage (between 0.0 and 1.0) of index documents which
contain a term, after which the word is considered to be a stop word
IOException - Can be thrown while reading from the IndexReader
public QueryAutoStopWordAnalyzer(Version matchVersion,
Analyzer delegate,
IndexReader indexReader,
Collection<String> fields,
int maxDocFreq)
throws IOException
matchVersion - Version to be used in StopFilterdelegate - Analyzer whose TokenStream will be filteredindexReader - IndexReader to identify the stopwords fromfields - Selection of fields to calculate stopwords formaxDocFreq - Document frequency terms should be above in order to be stopwords
IOException - Can be thrown while reading from the IndexReader| Method Detail |
|---|
@Deprecated
public int addStopWords(IndexReader reader)
throws IOException
QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader)
reader - The IndexReader which will be consulted to identify potential stop words that
exceed the required document frequency
IOException
@Deprecated
public int addStopWords(IndexReader reader,
int maxDocFreq)
throws IOException
QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, int)
reader - The IndexReader which will be consulted to identify potential stop words that
exceed the required document frequencymaxDocFreq - The maximum number of index documents which can contain a term, after which
the term is considered to be a stop word
IOException
@Deprecated
public int addStopWords(IndexReader reader,
float maxPercentDocs)
throws IOException
QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, float)
reader - The IndexReader which will be consulted to identify potential stop words that
exceed the required document frequencymaxPercentDocs - The maximum percentage (between 0.0 and 1.0) of index documents which
contain a term, after which the word is considered to be a stop word.
IOException
@Deprecated
public int addStopWords(IndexReader reader,
String fieldName,
float maxPercentDocs)
throws IOException
QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, Collection, float)
reader - The IndexReader which will be consulted to identify potential stop words that
exceed the required document frequencyfieldName - The field for which stopwords will be addedmaxPercentDocs - The maximum percentage (between 0.0 and 1.0) of index documents which
contain a term, after which the word is considered to be a stop word.
IOException
@Deprecated
public int addStopWords(IndexReader reader,
String fieldName,
int maxDocFreq)
throws IOException
QueryAutoStopWordAnalyzer(Version, Analyzer, IndexReader, Collection, int)
reader - The IndexReader which will be consulted to identify potential stop words that
exceed the required document frequencyfieldName - The field for which stopwords will be addedmaxDocFreq - The maximum number of index documents which
can contain a term, after which the term is considered to be a stop word.
IOException
public TokenStream tokenStream(String fieldName,
Reader reader)
Analyzer
tokenStream in class Analyzer
public TokenStream reusableTokenStream(String fieldName,
Reader reader)
throws IOException
Analyzer
reusableTokenStream in class AnalyzerIOExceptionpublic String[] getStopWords(String fieldName)
fieldName - The field for which stop words identified in "addStopWords"
method calls will be returned
public Term[] getStopWords()
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||