org.apache.lucene.analysis

Class LookaheadTokenFilter<T extends LookaheadTokenFilter.Position>

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.apache.lucene.analysis.LookaheadTokenFilter<T>

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

MockGraphTokenFilter, MockRandomLookaheadTokenFilter

public abstract class LookaheadTokenFilter<T extends LookaheadTokenFilter.Position>
extends TokenFilter

An abstract TokenFilter to make it easier to build graph token filters requiring some lookahead. This class handles the details of buffering up tokens, recording them by position, restoring them, providing access to them, etc.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	LookaheadTokenFilter.Position Holds all state for a single position; subclass this to record other state at each position.

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	end
protected int	inputPos
protected OffsetAttribute	offsetAtt
protected int	outputPos
protected PositionIncrementAttribute	posIncAtt
protected RollingBuffer<T>	positions
protected PositionLengthAttribute	posLenAtt

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	LookaheadTokenFilter(TokenStream input)

Method Summary

Modifier and Type	Method and Description
protected void	afterPosition() This is called when all input tokens leaving a given position have been returned.
protected void	insertToken() Call this only from within afterPosition, to insert a new token.
protected abstract T	newPosition()
protected boolean	nextToken() Call this when you are done looking ahead; it will set the next token to return.
protected boolean	peekToken() Returns true if there is a new token.
void	reset() This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end

Methods inherited from class org.apache.lucene.analysis.TokenStream

incrementToken

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

posIncAtt

protected final PositionIncrementAttribute posIncAtt

posLenAtt

protected final PositionLengthAttribute posLenAtt

offsetAtt

protected final OffsetAttribute offsetAtt

inputPos

protected int inputPos

outputPos

protected int outputPos

end

protected boolean end

positions

protected final RollingBuffer<T extends LookaheadTokenFilter.Position> positions

Constructor Detail

LookaheadTokenFilter

protected LookaheadTokenFilter(TokenStream input)

Method Detail

insertToken

protected void insertToken()
                    throws java.io.IOException

Call this only from within afterPosition, to insert a new token. After calling this you should set any necessary token you need.

Throws:

java.io.IOException

afterPosition

protected void afterPosition()
                      throws java.io.IOException

This is called when all input tokens leaving a given position have been returned. Override this and call insertToken and then set whichever token's attributes you want, if you want to inject a token starting from this position.

Throws:

java.io.IOException

newPosition

protected abstract T newPosition()

peekToken

protected boolean peekToken()
                     throws java.io.IOException

Returns true if there is a new token.

Throws:

java.io.IOException

nextToken

protected boolean nextToken()
                     throws java.io.IOException

Call this when you are done looking ahead; it will set the next token to return. Return the boolean back to the caller.

Throws:

java.io.IOException

reset

public void reset()
           throws java.io.IOException

Description copied from class: TokenFilter

This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).

NOTE: The default implementation chains the call to the input TokenStream, so be sure to call super.reset() when overriding this method.

Overrides:

reset in class TokenFilter

Throws:

java.io.IOException