org.apache.lucene.analysis

Class MockTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.MockTokenizer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class MockTokenizer
extends org.apache.lucene.analysis.Tokenizer

Tokenizer for testing.

This tokenizer is a replacement for WHITESPACE, SIMPLE, and KEYWORD tokenizers. If you are writing a component such as a TokenFilter, its a great idea to test it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:

• An internal state-machine is used for checking consumer consistency. These checks can be disabled with setEnableChecks(boolean).
• For convenience, optionally lowercases terms that it outputs.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_TOKEN_LENGTH
static int	KEYWORD Acts Similar to KeywordTokenizer.
static int	SIMPLE Acts like LetterTokenizer.
static int	WHITESPACE Acts Similar to WhitespaceTokenizer

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Constructor Summary

Constructors

Constructor and Description
MockTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader input, int pattern, boolean lowerCase, int maxTokenLength)
MockTokenizer(java.io.Reader input) Calls MockTokenizer(Reader, WHITESPACE, true)
MockTokenizer(java.io.Reader input, int pattern, boolean lowerCase)
MockTokenizer(java.io.Reader input, int pattern, boolean lowerCase, int maxTokenLength)

Method Summary

Modifier and Type	Method and Description
void	close()
void	end()
boolean	incrementToken()
protected boolean	isTokenChar(int c)
protected int	normalize(int c)
protected int	readCodePoint()
void	reset()
void	reset(java.io.Reader input)
void	setEnableChecks(boolean enableChecks) Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

WHITESPACE

public static final int WHITESPACE

Acts Similar to WhitespaceTokenizer

See Also:

Constant Field Values

KEYWORD

public static final int KEYWORD

Acts Similar to KeywordTokenizer. TODO: Keyword returns an "empty" token for an empty reader...

See Also:

Constant Field Values

SIMPLE

public static final int SIMPLE

Acts like LetterTokenizer.

See Also:

Constant Field Values

DEFAULT_MAX_TOKEN_LENGTH

public static final int DEFAULT_MAX_TOKEN_LENGTH

See Also:

Constant Field Values

Constructor Detail

MockTokenizer

public MockTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
                     java.io.Reader input,
                     int pattern,
                     boolean lowerCase,
                     int maxTokenLength)

MockTokenizer

public MockTokenizer(java.io.Reader input,
                     int pattern,
                     boolean lowerCase,
                     int maxTokenLength)

MockTokenizer

public MockTokenizer(java.io.Reader input,
                     int pattern,
                     boolean lowerCase)

MockTokenizer

public MockTokenizer(java.io.Reader input)

Calls MockTokenizer(Reader, WHITESPACE, true)

Method Detail

incrementToken

public final boolean incrementToken()
                             throws java.io.IOException

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

java.io.IOException

readCodePoint

protected int readCodePoint()
                     throws java.io.IOException

Throws:

java.io.IOException

isTokenChar

protected boolean isTokenChar(int c)

normalize

protected int normalize(int c)

reset

public void reset()
           throws java.io.IOException

Overrides:

reset in class org.apache.lucene.analysis.TokenStream

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class org.apache.lucene.analysis.Tokenizer

Throws:

java.io.IOException

reset

public void reset(java.io.Reader input)
           throws java.io.IOException

Overrides:

reset in class org.apache.lucene.analysis.Tokenizer

Throws:

java.io.IOException

end

public void end()
         throws java.io.IOException

Overrides:

end in class org.apache.lucene.analysis.TokenStream

Throws:

java.io.IOException

setEnableChecks

public void setEnableChecks(boolean enableChecks)

Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.