Uses of Package
sjm.parse.tokens

Packages that use sjm.parse.tokens

sjm.examples.coffee
sjm.examples.logic
sjm.examples.mechanics
sjm.examples.query
sjm.examples.reserved
sjm.examples.sling
sjm.examples.tokens
sjm.parse.tokens

Classes in sjm.parse.tokens used by sjm.examples.coffee

Tokenizer
A tokenizer divides a string into tokens.

Classes in sjm.parse.tokens used by sjm.examples.logic

TokenString
A TokenString is like a String, but it is a series of Tokens rather than a series of chars.

Classes in sjm.parse.tokens used by sjm.examples.mechanics

Word
A Word matches a word from a token assembly.

Classes in sjm.parse.tokens used by sjm.examples.query

TokenTester

Classes in sjm.parse.tokens used by sjm.examples.reserved

Token
A token represents a logical chunk of a string.

Tokenizer
A tokenizer divides a string into tokens.

TokenizerState
A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader.

TokenType
Objects of this class represent a type of token, such as "number" or "word".

WordState
A wordState returns a word from a reader.

Classes in sjm.parse.tokens used by sjm.examples.sling

Token
A token represents a logical chunk of a string.

TokenAssembly
A TokenAssembly is an Assembly whose elements are Tokens.

Tokenizer
A tokenizer divides a string into tokens.

TokenTester

Classes in sjm.parse.tokens used by sjm.examples.tokens

NumberState
A NumberState object returns a number from a reader.

Token
A token represents a logical chunk of a string.

Tokenizer
A tokenizer divides a string into tokens.

TokenizerState
A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader.

Classes in sjm.parse.tokens used by sjm.parse.tokens

Literal
A Literal matches a specific String from an assembly.

NumberState
A NumberState object returns a number from a reader.

QuoteState
A quoteState returns a quoted string token from a reader.

SlashSlashState
A slashSlash state ignores everything up to an end-of-line and returns the tokenizer's next token.

SlashStarState
A slashStar state ignores everything up to a closing star and slash, and then returns the tokenizer's next token.

SlashState
This state will either delegate to a comment-handling state, or return a token with just a slash in it.

SymbolNode
A SymbolNode object is a member of a tree that contains all possible prefixes of allowable symbols.

SymbolState
The idea of a symbol is a character that stands on its own, such as an ampersand or a parenthesis.

Token
A token represents a logical chunk of a string.

Tokenizer
A tokenizer divides a string into tokens.

TokenizerState
A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader.

TokenString
A TokenString is like a String, but it is a series of Tokens rather than a series of chars.

TokenType
Objects of this class represent a type of token, such as "number" or "word".

WhitespaceState
A whitespace state ignores whitespace (such as blanks and tabs), and returns the tokenizer's next token.

WordState
A wordState returns a word from a reader.