Tokenizer
in package
Table of Contents
Properties
- $parsers : array<string|int, Parser>
- The list of parsers to handle different CSS chars.
- $tokenizerContext : TokenizerContext
Methods
- tokenize() : Generator<string|int, Token|LintError>
- Tokenizes a CSS stream resource into an array of tokens.
- assertBlockTokenIsClean() : Generator<string|int, LintError>
- Assert that the block token is clean, meaning that it has a property token.
- assertTokenizerContextIsClean() : Generator<string|int, LintError|BlockToken>
- Assert that the tokenizer context is clean, meaning that the current token is closed.
- findNextCompatibleParser() : Parser|null
- Find the next parser that can handle tokens that the current token can transition to
- getParsers() : array<string|int, Parser>
- Returns the list of parsers used to handle different CSS chars.
- parseCharacter() : Generator<string|int, Token|LintError>
- Parses the current context of the tokenizer and yields tokens or lint errors.
- processBuffer() : Generator<string|int, Token|LintError>
- Process a buffer of characters and generate tokens.
- processCharacter() : Generator<string|int, Token|LintError>
- Process a single character and generate tokens if applicable.
- processStreamContent() : Generator<string|int, Token|LintError>
- Process the content of the stream chunk by chunk.
- resetTokenizerContext() : self
Properties
$parsers
The list of parsers to handle different CSS chars.
private
array<string|int, Parser>
$parsers
$tokenizerContext
private
TokenizerContext
$tokenizerContext
Methods
tokenize()
Tokenizes a CSS stream resource into an array of tokens.
public
tokenize(resource $stream) : Generator<string|int, Token|LintError>
Parameters
- $stream : resource
-
The CSS stream resource to tokenize.
Return values
Generator<string|int, Token|LintError> —An array of tokens or issues found during tokenizing.
assertBlockTokenIsClean()
Assert that the block token is clean, meaning that it has a property token.
private
assertBlockTokenIsClean(BlockToken $blockToken) : Generator<string|int, LintError>
Parameters
- $blockToken : BlockToken
Return values
Generator<string|int, LintError>assertTokenizerContextIsClean()
Assert that the tokenizer context is clean, meaning that the current token is closed.
private
assertTokenizerContextIsClean() : Generator<string|int, LintError|BlockToken>
Yields the current token if it is a block token for linting purposes.
Return values
Generator<string|int, LintError|BlockToken>findNextCompatibleParser()
Find the next parser that can handle tokens that the current token can transition to
private
findNextCompatibleParser(TokenBoundary $token) : Parser|null
Parameters
- $token : TokenBoundary
Return values
Parser|nullgetParsers()
Returns the list of parsers used to handle different CSS chars.
private
getParsers() : array<string|int, Parser>
Return values
array<string|int, Parser> —The list of parsers.
parseCharacter()
Parses the current context of the tokenizer and yields tokens or lint errors.
private
parseCharacter(Parser $parser) : Generator<string|int, Token|LintError>
Parameters
- $parser : Parser
Return values
Generator<string|int, Token|LintError>processBuffer()
Process a buffer of characters and generate tokens.
private
processBuffer(string $buffer, int &$position) : Generator<string|int, Token|LintError>
Parameters
- $buffer : string
-
The buffer to process
- $position : int
-
Current position in the buffer, passed by reference
Return values
Generator<string|int, Token|LintError>processCharacter()
Process a single character and generate tokens if applicable.
private
processCharacter(string $char) : Generator<string|int, Token|LintError>
Parameters
- $char : string
-
The character to process
Return values
Generator<string|int, Token|LintError>processStreamContent()
Process the content of the stream chunk by chunk.
private
processStreamContent(resource $stream) : Generator<string|int, Token|LintError>
Parameters
- $stream : resource
Return values
Generator<string|int, Token|LintError>resetTokenizerContext()
private
resetTokenizerContext() : self