- Direct Known Subclasses:
- HtmlDecorator, InlineDocTokenizer, JavaAnnotationTokenizer, JavaConstantTokenizer, LiteralTokenizer, MultilinesDocTokenizer, NotThreadSafeTokenizer, StringTokenizer
public abstract class Tokenizer
extends Channel<HtmlCodeBuilder>
A token highlighter is in charge to optionally consume the next characters (token) of a CodeReader
and to add the highlighted token to an HtmlCodeBuilder.
IMPORTANT: a Tokenizer must be THREAD-SAFE otherwise see NotThreadSafeTokenizer