antlr

Class TokenBuffer

public class TokenBuffer extends Object

A Stream of Token objects fed to the parser from a Tokenizer that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

See Also: Token antlr.Tokenizer TokenQueue

Field Summary
protected TokenStreaminput
intmarkerOffset
intnMarkers
intnumToConsume
TokenQueuequeue
Constructor Summary
TokenBuffer(TokenStream input_)
Create a token buffer
Method Summary
voidconsume()
Mark another token for deferred consumption
TokenStreamgetInput()
return the Tokenizer (needed by ParseView)
intLA(int i)
Get a lookahead token value
TokenLT(int i)
Get a lookahead token
intmark()
Return an integer marker that can be used to rewind the buffer to its current state.
voidreset()
Reset the input buffer to empty state
voidrewind(int mark)
Rewind the token buffer to a marker.

Field Detail

input

protected TokenStream input

markerOffset

int markerOffset

nMarkers

int nMarkers

numToConsume

int numToConsume

queue

TokenQueue queue

Constructor Detail

TokenBuffer

public TokenBuffer(TokenStream input_)
Create a token buffer

Method Detail

consume

public final void consume()
Mark another token for deferred consumption

getInput

public TokenStream getInput()
return the Tokenizer (needed by ParseView)

LA

public final int LA(int i)
Get a lookahead token value

LT

public final Token LT(int i)
Get a lookahead token

mark

public final int mark()
Return an integer marker that can be used to rewind the buffer to its current state.

reset

public final void reset()
Reset the input buffer to empty state

rewind

public final void rewind(int mark)
Rewind the token buffer to a marker.

Parameters: mark Marker returned previously from mark()