sjm.parse.tokens
Class TokenizerState

java.lang.Object
  |
  +--sjm.parse.tokens.TokenizerState

Direct Known Subclasses:

NumberState, QuoteState, SlashSlashState, SlashStarState, SlashState, SymbolState, WhitespaceState, WordState

public abstract class TokenizerState

extends java.lang.Object

A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader. The tokenizer will typically have a character state table that decides which state to use, depending on an initial character. If a single character is insufficient, a state such as SlashState will read a second character, and may delegate to another state, such as SlashStarState. This prospect of delegation is the reason that the nextToken() method has a tokenizer argument.

Constructor Summary

TokenizerState()

Method Summary

abstract Token

nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a token that represents a logical piece of a reader.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerState

public TokenizerState()

Method Detail

nextToken

public abstract Token nextToken(java.io.PushbackReader r,
                                int c,
                                Tokenizer t)
                         throws java.io.IOException

Return a token that represents a logical piece of a reader.

Parameters:

PushbackReader - a reader to read from

c - the character that a tokenizer used to determine to use this state

Tokenizer - the tokenizer conducting the overall tokenization of the reader

Returns:

a token that represents a logical piece of the reader

Throws:

java.io.IOException - if there is any problem reading