Uses of Class
sjm.parse.tokens.Tokenizer

Packages that use Tokenizer

sjm.examples.coffee
sjm.examples.reserved
sjm.examples.sling
sjm.examples.tokens
sjm.parse.tokens

Uses of Tokenizer in sjm.examples.coffee

Methods in sjm.examples.coffee that return Tokenizer

static Tokenizer

CoffeeParser.tokenizer() Returns a tokenizer that allows spaces to appear inside the "words" that identify a coffee's name.

Uses of Tokenizer in sjm.examples.reserved

Methods in sjm.examples.reserved that return Tokenizer

static Tokenizer

VolumeQuery2.tokenizer() Return a customized tokenizer that uses WordOrReservedState in place of WordState.

Methods in sjm.examples.reserved with parameters of type Tokenizer

Token

WordOrReservedState.nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a reserved token or a word token from a reader.

Uses of Tokenizer in sjm.examples.sling

Fields in sjm.examples.sling declared as Tokenizer

protected Tokenizer

SlingParser.tokenizer

Methods in sjm.examples.sling that return Tokenizer

Tokenizer

SlingParser.tokenizer() Creates a tokenizer that uses a WordOrReservedState instead of a normal WordState.

Uses of Tokenizer in sjm.examples.tokens

Methods in sjm.examples.tokens with parameters of type Tokenizer

protected Token

ScientificNumberState.value(java.io.PushbackReader r, Tokenizer t)

Uses of Tokenizer in sjm.parse.tokens

Fields in sjm.parse.tokens declared as Tokenizer

protected Tokenizer

TokenStringSource.tokenizer

Methods in sjm.parse.tokens with parameters of type Tokenizer

abstract Token	TokenizerState.nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a token that represents a logical piece of a reader.
Token	QuoteState.nextToken(java.io.PushbackReader r, int cin, Tokenizer t) Return a quoted string token from a reader.
Token	NumberState.nextToken(java.io.PushbackReader r, int cin, Tokenizer t) Return a number token from a reader.
protected Token	NumberState.value(java.io.PushbackReader r, Tokenizer t)
Token	SlashStarState.nextToken(java.io.PushbackReader r, int theStar, Tokenizer t) Ignore everything up to a closing star and slash, and then return the tokenizer's next token.
Token	SlashState.nextToken(java.io.PushbackReader r, int theSlash, Tokenizer t) Either delegate to a comment-handling state, or return a token with just a slash in it.
Token	SlashSlashState.nextToken(java.io.PushbackReader r, int theSlash, Tokenizer t) Ignore everything up to an end-of-line and return the tokenizer's next token.
Token	SymbolState.nextToken(java.io.PushbackReader r, int first, Tokenizer t) Return a symbol token from a reader.
Token	WordState.nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a word token from a reader.
Token	WhitespaceState.nextToken(java.io.PushbackReader r, int aWhitespaceChar, Tokenizer t) Ignore whitespace (such as blanks and tabs), and return the tokenizer's next token.

Constructors in sjm.parse.tokens with parameters of type Tokenizer

TokenStringSource.TokenStringSource(Tokenizer tokenizer, java.lang.String delimiter)
Constructs a TokenStringSource that will read TokenStrings using the specified tokenizer, delimited by the specified delimiter.

TokenString.TokenString(Tokenizer t)
Constructs a tokenString from the supplied reader and tokenizer.

TokenAssembly.TokenAssembly(Tokenizer t)
Constructs a TokenAssembly on a TokenString constructed from the given Tokenizer.