# Class Tokenizer

java.lang.Object
com.codename1.ui.editor.Tokenizer

public final class Tokenizer extends Object

A small, allocation light, stateful lexer for the pure code editor. It tokenizes a single line at a time given the lexer state carried over from the previous line, and returns the state to carry into the next line. This makes rehighlighting incremental: after an edit only the changed line and any following lines whose entry state changed need to be retokenized.

Only the spans that carry color (keyword, string, comment, number) are emitted; the gaps between them are drawn in the default text color.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static final class`
  `Tokenizer.Token`
  A colored span within a line: `[start, start + length)` has the given kind.
  `static final class`
  `Tokenizer.TokenLine`
  The result of tokenizing one line: the colored spans and the lexer state to carry to the next line.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `COMMENT`
  Token kind: a comment.
  `static final int`
  `KEYWORD`
  Token kind: a language keyword.
  `static final int`
  `NUMBER`
  Token kind: a numeric literal.
  `static final int`
  `STATE_BLOCK_COMMENT`
  Lexer state: inside a block comment that opened on a previous line.
  `static final int`
  `STATE_NORMAL`
  Lexer state: normal code.
  `static final int`
  `STATE_TEMPLATE`
  Lexer state: inside a backtick template string that opened on a previous line.
  `static final int`
  `STRING`
  Token kind: a string or character literal.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Tokenizer(LanguageDef def)`
  Creates a tokenizer for the given language.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `Tokenizer.TokenLine`
  `tokenize(String line, int startState)`
  Tokenizes one line.

  ### Methods inherited from class Object

  `clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### KEYWORD

    public static final int KEYWORD
    Token kind: a language keyword.
    See Also:
      - Constant Field Values

  - ### STRING

    public static final int STRING
    Token kind: a string or character literal.
    See Also:
      - Constant Field Values

  - ### COMMENT

    public static final int COMMENT
    Token kind: a comment.
    See Also:
      - Constant Field Values

  - ### NUMBER

    public static final int NUMBER
    Token kind: a numeric literal.
    See Also:
      - Constant Field Values

  - ### STATE_NORMAL

    public static final int STATE_NORMAL
    Lexer state: normal code.
    See Also:
      - Constant Field Values

  - ### STATE_BLOCK_COMMENT

    public static final int STATE_BLOCK_COMMENT
    Lexer state: inside a block comment that opened on a previous line.
    See Also:
      - Constant Field Values

  - ### STATE_TEMPLATE

    public static final int STATE_TEMPLATE
    Lexer state: inside a backtick template string that opened on a previous line.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### Tokenizer

    public Tokenizer(LanguageDef def)
    Creates a tokenizer for the given language.

- ## Method Details

  - ### tokenize

    public Tokenizer.TokenLine tokenize(String line, int startState)

    Tokenizes one line.

    ###### Parameters

    - `line`: the line text (without its trailing newline)

    - `startState`: the lexer state carried from the previous line

    ###### Returns

    the colored spans and exit state