org.apache.lucene.analysis.ngram
Class NGramTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.ngram.NGramTokenizer
- All Implemented Interfaces:
- Closeable
public final class NGramTokenizer
- extends org.apache.lucene.analysis.Tokenizer
Tokenizes the input into n-grams of the given size(s).
| Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State |
| Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
|
Constructor Summary |
NGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
Reader input,
int minGram,
int maxGram)
Creates NGramTokenizer with given min and max n-grams. |
NGramTokenizer(org.apache.lucene.util.AttributeSource source,
Reader input,
int minGram,
int maxGram)
Creates NGramTokenizer with given min and max n-grams. |
NGramTokenizer(Reader input)
Creates NGramTokenizer with default min and max n-grams. |
NGramTokenizer(Reader input,
int minGram,
int maxGram)
Creates NGramTokenizer with given min and max n-grams. |
|
Method Summary |
void |
end()
|
boolean |
incrementToken()
Returns the next token in the stream, or null at EOS. |
void |
reset()
|
| Methods inherited from class org.apache.lucene.analysis.Tokenizer |
close, correctOffset, reset |
| Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString |
DEFAULT_MIN_NGRAM_SIZE
public static final int DEFAULT_MIN_NGRAM_SIZE
- See Also:
- Constant Field Values
DEFAULT_MAX_NGRAM_SIZE
public static final int DEFAULT_MAX_NGRAM_SIZE
- See Also:
- Constant Field Values
NGramTokenizer
public NGramTokenizer(Reader input,
int minGram,
int maxGram)
- Creates NGramTokenizer with given min and max n-grams.
- Parameters:
input - Reader holding the input to be tokenizedminGram - the smallest n-gram to generatemaxGram - the largest n-gram to generate
NGramTokenizer
public NGramTokenizer(org.apache.lucene.util.AttributeSource source,
Reader input,
int minGram,
int maxGram)
- Creates NGramTokenizer with given min and max n-grams.
- Parameters:
source - AttributeSource to useinput - Reader holding the input to be tokenizedminGram - the smallest n-gram to generatemaxGram - the largest n-gram to generate
NGramTokenizer
public NGramTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
Reader input,
int minGram,
int maxGram)
- Creates NGramTokenizer with given min and max n-grams.
- Parameters:
factory - AttributeSource.AttributeFactory to useinput - Reader holding the input to be tokenizedminGram - the smallest n-gram to generatemaxGram - the largest n-gram to generate
NGramTokenizer
public NGramTokenizer(Reader input)
- Creates NGramTokenizer with default min and max n-grams.
- Parameters:
input - Reader holding the input to be tokenized
incrementToken
public boolean incrementToken()
throws IOException
- Returns the next token in the stream, or null at EOS.
- Specified by:
incrementToken in class org.apache.lucene.analysis.TokenStream
- Throws:
IOException
end
public void end()
- Overrides:
end in class org.apache.lucene.analysis.TokenStream
reset
public void reset()
throws IOException
- Overrides:
reset in class org.apache.lucene.analysis.TokenStream
- Throws:
IOException