Package io.github.qishr.cascara.common.lang.processor

Interface Tokenizer

java.lang.Object
        io.github.qishr.cascara.common.lang.processor.Tokenizer

All Implemented Interfaces:
    Processor

All Known Implementing Classes:
    JsonTokenizer, XmlTokenizer, YamlTokenizer

---

public interface Tokenizer

Method Summary

Modifier and Type	Method	Description
public default List	tokenize(String text)	High-level API: Tokenizes a complete String eagerly.
public default List	tokenize(InputStream is)	High-level API: Tokenizes an entire InputStream eagerly.
public abstract void	open(String text)	Low-level Streaming API: Resets the tokenizer state to read from a String.
public abstract void	open(Reader reader)
public abstract void	open(InputStream is)	Low-level Streaming API: Resets the tokenizer state to read from a stream.
public abstract T	nextToken()	Low-level Streaming API: Pulls the next token on demand.
public default Set<? extends TokenType>	getTokenTypes()
public abstract int	getOffset()

Method Details

tokenize

public default List tokenize(String text)

High-level API: Tokenizes a complete String eagerly.

---

tokenize

public default List tokenize(InputStream is)

High-level API: Tokenizes an entire InputStream eagerly.

---

open

public abstract void open(String text)

Low-level Streaming API: Resets the tokenizer state to read from a String.

---

open

public abstract void open(Reader reader)

---

open

public abstract void open(InputStream is)

Low-level Streaming API: Resets the tokenizer state to read from a stream.

---

nextToken

public abstract T nextToken()

Low-level Streaming API: Pulls the next token on demand.

---

getTokenTypes

public default Set<? extends TokenType> getTokenTypes()

---

getOffset

public abstract int getOffset()

---