AOWP


aowp.parser.parser.PHP_Parser_Tokenizer
/parser/Parser/Tokenizer.php at line 54

Class PHP_Parser_Tokenizer

PHP_Parser_Tokenizer

public class PHP_Parser_Tokenizer

The tokenizer wrapper for parser - implements the 'standard?' yylex interface

2 main methods:

Version:
$Id: Tokenizer.php,v 1.13 2007/03/05 05:05:50 cellog Exp $

Field Summary
int

$N

Total Number of tokens.

private mixed

$_docParser

private string

$_globalSearch

String of global variable to search forphpDocumentor-specific usage, extracted from documentation's @global tag

private mixed

$_options

Tokenizing options

private mixed

$_trackingWhitespace

private mixed

$_whitespace

boolean

$debug

Debugging on/off

string

$lastComment

The string of the last Comment Token

int

$lastCommentLine

ID of the last Comment Token

int

$lastCommentToken

ID of the last Comment Token

string

$lastParsedComment

The string of the last Comment Token

int

$line

Current line.

int

$pos

Current token position.

int

$token

The current token (either a ord(';') or token numer - see php tokenizer.

mixed

$tokenValue

array

$tokens

Tokens - array of all the tokens.

string

$value

The value associated with a token - eg.

string

$valueWithWhitespace

The value associated with a token plus preceding whitespace, if anyThis is only filled if whitespace attachment is turned on, for performance reasons

Constructor Summary

PHP_Parser_Tokenizer(mixed data, mixed docparser, string PHP)

ConstructorLoad the tokenizer - with a string to tokenize.

Method Summary
private void

_handleDocumentation()

Helper function for advance(), parses and publishes doc comments as necessary

private void

_setGlobalSearch(mixed var, string )

boolean

advance()

The main advance call required by the parser return true if a token is available, false if no more are available.

array

getLastComment()

get the last comment block (and reset it)

void

getValue()

string

getWhitespace()

Return the whitespace (if any) that preceded the current token

void

globalSearch(mixed var, string global)

Compare global variable to search value, to see if we've found a variable that must be documented

void

haltLexing()

void

handleMessage(mixed msg, MsgServer_Msg )

string

parseError()

return something useful, when a parse error occurs.

void

releaseInstance()

void

stopTrackingWhitespace()

void

trackWhitespace()

Field Detail

/parser/Parser/Tokenizer.php at line 78

N

public int $N

Total Number of tokens.


/parser/Parser/Tokenizer.php at line 174

_docParser

private mixed $_docParser

/parser/Parser/Tokenizer.php at line 165

_globalSearch

private string $_globalSearch = false

String of global variable to search for

phpDocumentor-specific usage, extracted from documentation's @global tag


/parser/Parser/Tokenizer.php at line 171

_options

private mixed $_options

Tokenizing options


/parser/Parser/Tokenizer.php at line 173

_trackingWhitespace

private mixed $_trackingWhitespace

/parser/Parser/Tokenizer.php at line 172

_whitespace

private mixed $_whitespace

/parser/Parser/Tokenizer.php at line 64

debug

public boolean $debug = false

Debugging on/off


/parser/Parser/Tokenizer.php at line 146

lastComment

public string $lastComment

The string of the last Comment Token


/parser/Parser/Tokenizer.php at line 137

lastCommentLine

public int $lastCommentLine

ID of the last Comment Token


/parser/Parser/Tokenizer.php at line 128

lastCommentToken

public int $lastCommentToken

ID of the last Comment Token


/parser/Parser/Tokenizer.php at line 155

lastParsedComment

public string $lastParsedComment

The string of the last Comment Token


/parser/Parser/Tokenizer.php at line 85

line

public int $line

Current line.


/parser/Parser/Tokenizer.php at line 92

pos

public int $pos = -1

Current token position.


/parser/Parser/Tokenizer.php at line 100

token

public int $token

The current token (either a ord(';') or token numer - see php tokenizer.


/parser/Parser/Tokenizer.php at line 56

tokenValue

public mixed $tokenValue

/parser/Parser/Tokenizer.php at line 71

tokens

public array $tokens

Tokens - array of all the tokens.


/parser/Parser/Tokenizer.php at line 109

value

public string $value

The value associated with a token - eg. for T_STRING it's the string


/parser/Parser/Tokenizer.php at line 119

valueWithWhitespace

public string $valueWithWhitespace

The value associated with a token plus preceding whitespace, if any

This is only filled if whitespace attachment is turned on, for performance reasons


Constructor Detail

/parser/Parser/Tokenizer.php at line 188

PHP_Parser_Tokenizer

public PHP_Parser_Tokenizer(mixed data, mixed docparser, string PHP)

Constructor

Load the tokenizer - with a string to tokenize. tidies up array, sets vars pos, line, N and tokens

Parameters:
PHP - code to serialize

Method Detail

/parser/Parser/Tokenizer.php at line 298

_handleDocumentation

private void _handleDocumentation()

Helper function for advance(), parses and publishes doc comments as necessary


/parser/Parser/Tokenizer.php at line 237

_setGlobalSearch

private void _setGlobalSearch(mixed var, string )


/parser/Parser/Tokenizer.php at line 327

advance

public boolean advance()

The main advance call required by the parser

return true if a token is available, false if no more are available. skips stuff that is not a valid token stores lastcomment, lastcommenttoken

Returns:
- true = have tokens

/parser/Parser/Tokenizer.php at line 279

getLastComment

public array getLastComment()

get the last comment block (and reset it)

Returns:
array('comment' => $commmentstring, 'parsed' => $parsed, 'line' => $linenumber)

/parser/Parser/Tokenizer.php at line 429

getValue

public void getValue()

/parser/Parser/Tokenizer.php at line 218

getWhitespace

public string getWhitespace()

Return the whitespace (if any) that preceded the current token


/parser/Parser/Tokenizer.php at line 257

globalSearch

public void globalSearch(mixed var, string global)

Compare global variable to search value, to see if we've found a variable that must be documented

Parameters:
global - variable found in source code

/parser/Parser/Tokenizer.php at line 208

haltLexing

public void haltLexing()

/parser/Parser/Tokenizer.php at line 226

handleMessage

public void handleMessage(mixed msg, MsgServer_Msg )


/parser/Parser/Tokenizer.php at line 442

parseError

public string parseError()

return something useful, when a parse error occurs.

used to build error messages if the parser fails, and needs to know the line number..


/parser/Parser/Tokenizer.php at line 448

releaseInstance

public void releaseInstance()

/parser/Parser/Tokenizer.php at line 247

stopTrackingWhitespace

public void stopTrackingWhitespace()

/parser/Parser/Tokenizer.php at line 242

trackWhitespace

public void trackWhitespace()

AOWP