Class Tokenizer

java.lang.Object
com.codename1.ui.editor.Tokenizer
public final class Tokenizer extends Object

A small, allocation light, stateful lexer for the pure code editor. It tokenizes a single line at a time given the lexer state carried over from the previous line, and returns the state to carry into the next line. This makes rehighlighting incremental: after an edit only the changed line and any following lines whose entry state changed need to be retokenized.

Only the spans that carry color (keyword, string, comment, number) are emitted; the gaps between them are drawn in the default text color.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static final class 
    Tokenizer.Token
    A colored span within a line: [start, start + length) has the given kind.
    static final class 
    Tokenizer.TokenLine
    The result of tokenizing one line: the colored spans and the lexer state to carry to the next line.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    COMMENT
    Token kind: a comment.
    static final int
    KEYWORD
    Token kind: a language keyword.
    static final int
    NUMBER
    Token kind: a numeric literal.
    static final int
    STATE_BLOCK_COMMENT
    Lexer state: inside a block comment that opened on a previous line.
    static final int
    STATE_NORMAL
    Lexer state: normal code.
    static final int
    STATE_TEMPLATE
    Lexer state: inside a backtick template string that opened on a previous line.
    static final int
    STRING
    Token kind: a string or character literal.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Tokenizer(LanguageDef def)
    Creates a tokenizer for the given language.

  • Method Summary

    Modifier and Type
    Method
    Description
    Tokenizer.TokenLine
    tokenize(String line, int startState)
    Tokenizes one line.

    Methods inherited from class Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • KEYWORD

      public static final int KEYWORD
      Token kind: a language keyword.
      See Also:

    • STRING

      public static final int STRING
      Token kind: a string or character literal.
      See Also:

    • COMMENT

      public static final int COMMENT
      Token kind: a comment.
      See Also:

    • NUMBER

      public static final int NUMBER
      Token kind: a numeric literal.
      See Also:

    • STATE_NORMAL

      public static final int STATE_NORMAL
      Lexer state: normal code.
      See Also:

    • STATE_BLOCK_COMMENT

      public static final int STATE_BLOCK_COMMENT
      Lexer state: inside a block comment that opened on a previous line.
      See Also:

    • STATE_TEMPLATE

      public static final int STATE_TEMPLATE
      Lexer state: inside a backtick template string that opened on a previous line.
      See Also:

  • Constructor Details

    • Tokenizer

      public Tokenizer(LanguageDef def)
      Creates a tokenizer for the given language.

  • Method Details

    • tokenize

      public Tokenizer.TokenLine tokenize(String line, int startState)

      Tokenizes one line.

      Parameters

      • line: the line text (without its trailing newline)

      • startState: the lexer state carried from the previous line

      Returns

      the colored spans and exit state