Class TTokenizer
Unit
Declaration
type TTokenizer = class(TObject)
Description
Converts an input TStream to a sequence of TToken objects.
Source: source/component/PasDoc_Tokenizer.pas (line 336).
Hierarchy
- TObject
- TTokenizer
Overview
Fields
| Protected | FOnMessage: TPasDocMessageEvent; |
| Protected | FVerbosity: Cardinal; |
| Protected | BufferedChar: Char; |
| Protected | EOS: Boolean; |
| Protected | IsCharBuffered: Boolean; |
| Protected | Line: Integer; |
| Protected | Stream: TStream; |
| Protected | FStreamName: string; |
| Protected | FStreamAbsoluteFileName: string; |
Methods
| Protected | procedure DoError(const AMessage: string; const AArguments: array of const); |
| Protected | procedure DoMessage(const AVerbosity: Cardinal; const MessageType: TPasDocMessageType; const AMessage: string; const AArguments: array of const); |
| Protected | procedure CheckForDirective(const t: TToken); |
| Protected | procedure ConsumeChar; |
| Protected | function CreateSymbolToken(const st: TSymbolType; const s: string): TToken; overload; |
| Protected | function CreateSymbolToken(const st: TSymbolType): TToken; overload; |
| Protected | function GetChar(out c: AnsiChar ): Integer; |
| Protected | function PeekChar(out c: Char): Boolean; |
| Protected | function ReadCommentType1: TToken; |
| Protected | function ReadCommentType2: TToken; |
| Protected | function ReadCommentType3: TToken; |
| Protected | function ReadAttAssemblerRegister: TToken; |
| Protected | function ReadLiteralString: TToken; |
| Protected | function ReadDoubleQuotedString: TToken; |
| Protected | function ReadToken(const StartChar: Char; const AllowedChars: TCharSet; const TokenType: TTokenType; out T: TToken): Boolean; |
| Protected | function ReadIdentifierToken(const StartChar: Char; out T: TToken): Boolean; |
| Public | constructor Create( const AStream: TStream; const OnMessageEvent: TPasDocMessageEvent; const VerbosityLevel: Cardinal; const AStreamName, AStreamAbsoluteFileName: string); |
| Public | destructor Destroy; override; |
| Public | function HasData: Boolean; |
| Public | function GetStreamInfo: string; |
| Public | function GetToken(const NilOnEnd: Boolean = false; const NilOnInvalidContent: Boolean = false): TToken; |
| Public | procedure UnGetToken(var T: TToken); |
| Public | function SkipUntilCompilerDirective: TToken; |
Properties
| Public | property OnMessage: TPasDocMessageEvent read FOnMessage write FOnMessage; |
| Public | property Verbosity: Cardinal read FVerbosity write FVerbosity; |
| Public | property StreamName: string read FStreamName; |
| Public | property StreamAbsoluteFileName: string read FStreamAbsoluteFileName; |
Description
Fields
| Protected | FOnMessage: TPasDocMessageEvent; |
|
This item has no description. | |
| Protected | FVerbosity: Cardinal; |
|
This item has no description. | |
| Protected | BufferedChar: Char; |
|
if IsCharBuffered is true, this field contains the buffered character | |
| Protected | EOS: Boolean; |
|
true if end of stream Stream has been reached, false otherwise | |
| Protected | IsCharBuffered: Boolean; |
|
if this is true, BufferedChar contains a buffered character; the next call to GetChar or PeekChar will return this character, not the next in the associated stream Stream | |
| Protected | Line: Integer; |
|
current line number in stream Stream; useful when giving error messages | |
| Protected | Stream: TStream; |
|
Input stream this tokenizer is working on. In case of STRING_UNICODE, all reads should go through the TStreamReader API, so underlying TStream is not exposed as a field even. | |
| Protected | FStreamName: string; |
|
This item has no description. | |
| Protected | FStreamAbsoluteFileName: string; |
|
This item has no description. | |
Methods
| Protected | procedure DoError(const AMessage: string; const AArguments: array of const); |
|
This item has no description. | |
| Protected | procedure DoMessage(const AVerbosity: Cardinal; const MessageType: TPasDocMessageType; const AMessage: string; const AArguments: array of const); |
|
This item has no description. | |
| Protected | procedure CheckForDirective(const t: TToken); |
|
This item has no description. | |
| Protected | procedure ConsumeChar; |
|
This item has no description. | |
| Protected | function CreateSymbolToken(const st: TSymbolType; const s: string): TToken; overload; |
|
This item has no description. | |
| Protected | function CreateSymbolToken(const st: TSymbolType): TToken; overload; |
|
Uses default symbol representation, from SymbolNames[st] | |
| Protected | function GetChar(out c: AnsiChar ): Integer; |
|
Read next character (WideChar in case of STRING_UNICODE). Returns how many bytes this character takes, or 0 if cannot read. | |
| Protected | function PeekChar(out c: Char): Boolean; |
|
This item has no description. | |
| Protected | function ReadCommentType1: TToken; |
|
This item has no description. | |
| Protected | function ReadCommentType2: TToken; |
|
This item has no description. | |
| Protected | function ReadCommentType3: TToken; |
|
This item has no description. | |
| Protected | function ReadAttAssemblerRegister: TToken; |
|
This item has no description. | |
| Protected | function ReadLiteralString: TToken; |
|
This item has no description. | |
| Protected | function ReadDoubleQuotedString: TToken; |
|
This item has no description. | |
| Protected | function ReadToken(const StartChar: Char; const AllowedChars: TCharSet; const TokenType: TTokenType; out T: TToken): Boolean; |
|
Read the rest of token of type TokenType, knowing the 1st character of this token is StartChar, and subsequent characters of this token are in set AllowedChars. Returns | |
| Protected | function ReadIdentifierToken(const StartChar: Char; out T: TToken): Boolean; |
|
Like ReadToken, but for identifiers. The set of valid subsequent characters is defined by IsIdentifierOtherChar function, and the token type is always TOK_IDENTIFIER. | |
| Public | constructor Create( const AStream: TStream; const OnMessageEvent: TPasDocMessageEvent; const VerbosityLevel: Cardinal; const AStreamName, AStreamAbsoluteFileName: string); |
|
Creates a TTokenizer and associates it with given input TStream. Note that AStream will be freed when this object will be freed. | |
| Public | destructor Destroy; override; |
|
Releases all dynamically allocated memory. | |
| Public | function HasData: Boolean; |
|
This item has no description. | |
| Public | function GetStreamInfo: string; |
|
This item has no description. | |
| Public | function GetToken(const NilOnEnd: Boolean = false; const NilOnInvalidContent: Boolean = false): TToken; |
|
Get next token from stream.
Parameters
| |
| Public | procedure UnGetToken(var T: TToken); |
|
Makes the token T next to be returned by GetToken. Also sets T to You cannot have more than one "unget" token. If you only call UnGetToken after some GetToken, you are safe. | |
| Public | function SkipUntilCompilerDirective: TToken; |
|
Skip all chars until it encounters some compiler directive, like $ELSE or $ENDIF. Returns either | |
Properties
| Public | property OnMessage: TPasDocMessageEvent read FOnMessage write FOnMessage; |
|
This item has no description. | |
| Public | property Verbosity: Cardinal read FVerbosity write FVerbosity; |
|
This item has no description. | |
| Public | property StreamName: string read FStreamName; |
|
Informative to user name of the stream from which this token was read. This can be a filename (relative or absolute, however user specified it), but it also can be something arbitrary like "$if / $elseif condition". So don't treat it like a reliable filename, for this use StreamAbsoluteFileName. | |
| Public | property StreamAbsoluteFileName: string read FStreamAbsoluteFileName; |
|
Filename, always absolute, of the underlying file of this stream. Empty ('') if this is not a file stream. | |
Generated by PasDoc 1.0.2.