Class TTokenizer
Unit
Declaration
type TTokenizer = class(TObject)
Description
Converts an input TStream to a sequence of TToken objects.
Source: source/component/PasDoc_Tokenizer.pas (line 325).
Hierarchy
- TObject
- TTokenizer
Overview
Fields
| Protected | FOnMessage: TPasDocMessageEvent; |
| Protected | FVerbosity: Cardinal; |
| Protected | BufferedChar: Char; |
| Protected | EOS: Boolean; |
| Protected | IsCharBuffered: Boolean; |
| Protected | Line: Integer; |
| Protected | Stream: TStream; |
| Protected | FStreamName: string; |
| Protected | FStreamAbsoluteFileName: string; |
Methods
| Protected | procedure DoError(const AMessage: string; const AArguments: array of const); |
| Protected | procedure DoMessage(const AVerbosity: Cardinal; const MessageType: TPasDocMessageType; const AMessage: string; const AArguments: array of const); |
| Protected | procedure CheckForDirective(const t: TToken); |
| Protected | procedure ConsumeChar; |
| Protected | function CreateSymbolToken(const st: TSymbolType; const s: string): TToken; overload; |
| Protected | function CreateSymbolToken(const st: TSymbolType): TToken; overload; |
| Protected | function GetChar(out c: AnsiChar): Integer; |
| Protected | function PeekChar(out c: Char): Boolean; |
| Protected | function ReadCommentType1: TToken; |
| Protected | function ReadCommentType2: TToken; |
| Protected | function ReadCommentType3: TToken; |
| Protected | function ReadAttAssemblerRegister: TToken; |
| Protected | function ReadLiteralString(var t: TToken): Boolean; |
| Protected | function ReadToken(c: Char; const s: TCharSet; const TT: TTokenType; var t: TToken): Boolean; |
| Public | constructor Create( const AStream: TStream; const OnMessageEvent: TPasDocMessageEvent; const VerbosityLevel: Cardinal; const AStreamName, AStreamAbsoluteFileName: string); |
| Public | destructor Destroy; override; |
| Public | function HasData: Boolean; |
| Public | function GetStreamInfo: string; |
| Public | function GetToken(const NilOnEnd: Boolean = false; const NilOnInvalidContent: Boolean = false): TToken; |
| Public | procedure UnGetToken(var T: TToken); |
| Public | function SkipUntilCompilerDirective: TToken; |
Properties
| Public | property OnMessage: TPasDocMessageEvent read FOnMessage write FOnMessage; |
| Public | property Verbosity: Cardinal read FVerbosity write FVerbosity; |
| Public | property StreamName: string read FStreamName; |
| Public | property StreamAbsoluteFileName: string read FStreamAbsoluteFileName; |
Description
Fields
| Protected | FOnMessage: TPasDocMessageEvent; |
|
This item has no description. | |
| Protected | FVerbosity: Cardinal; |
|
This item has no description. | |
| Protected | BufferedChar: Char; |
|
if IsCharBuffered is true, this field contains the buffered character | |
| Protected | EOS: Boolean; |
|
true if end of stream Stream has been reached, false otherwise | |
| Protected | IsCharBuffered: Boolean; |
|
if this is true, BufferedChar contains a buffered character; the next call to GetChar or PeekChar will return this character, not the next in the associated stream Stream | |
| Protected | Line: Integer; |
|
current line number in stream Stream; useful when giving error messages | |
| Protected | Stream: TStream; |
|
the input stream this tokenizer is working on | |
| Protected | FStreamName: string; |
|
This item has no description. | |
| Protected | FStreamAbsoluteFileName: string; |
|
This item has no description. | |
Methods
| Protected | procedure DoError(const AMessage: string; const AArguments: array of const); |
|
This item has no description. | |
| Protected | procedure DoMessage(const AVerbosity: Cardinal; const MessageType: TPasDocMessageType; const AMessage: string; const AArguments: array of const); |
|
This item has no description. | |
| Protected | procedure CheckForDirective(const t: TToken); |
|
This item has no description. | |
| Protected | procedure ConsumeChar; |
|
This item has no description. | |
| Protected | function CreateSymbolToken(const st: TSymbolType; const s: string): TToken; overload; |
|
This item has no description. | |
| Protected | function CreateSymbolToken(const st: TSymbolType): TToken; overload; |
|
Uses default symbol representation, from SymbolNames[st] | |
| Protected | function GetChar(out c: AnsiChar): Integer; |
|
Returns 1 on success or 0 on failure | |
| Protected | function PeekChar(out c: Char): Boolean; |
|
This item has no description. | |
| Protected | function ReadCommentType1: TToken; |
|
This item has no description. | |
| Protected | function ReadCommentType2: TToken; |
|
This item has no description. | |
| Protected | function ReadCommentType3: TToken; |
|
This item has no description. | |
| Protected | function ReadAttAssemblerRegister: TToken; |
|
This item has no description. | |
| Protected | function ReadLiteralString(var t: TToken): Boolean; |
|
This item has no description. | |
| Protected | function ReadToken(c: Char; const s: TCharSet; const TT: TTokenType; var t: TToken): Boolean; |
|
This item has no description. | |
| Public | constructor Create( const AStream: TStream; const OnMessageEvent: TPasDocMessageEvent; const VerbosityLevel: Cardinal; const AStreamName, AStreamAbsoluteFileName: string); |
|
Creates a TTokenizer and associates it with given input TStream. Note that AStream will be freed when this object will be freed. | |
| Public | destructor Destroy; override; |
|
Releases all dynamically allocated memory. | |
| Public | function HasData: Boolean; |
|
This item has no description. | |
| Public | function GetStreamInfo: string; |
|
This item has no description. | |
| Public | function GetToken(const NilOnEnd: Boolean = false; const NilOnInvalidContent: Boolean = false): TToken; |
|
Get next token from stream.
Parameters
| |
| Public | procedure UnGetToken(var T: TToken); |
|
Makes the token T next to be returned by GetToken. Also sets T to You cannot have more than one "unget" token. If you only call UnGetToken after some GetToken, you are safe. | |
| Public | function SkipUntilCompilerDirective: TToken; |
|
Skip all chars until it encounters some compiler directive, like $ELSE or $ENDIF. Returns either | |
Properties
| Public | property OnMessage: TPasDocMessageEvent read FOnMessage write FOnMessage; |
|
This item has no description. | |
| Public | property Verbosity: Cardinal read FVerbosity write FVerbosity; |
|
This item has no description. | |
| Public | property StreamName: string read FStreamName; |
|
Informative to user name of the stream from which this token was read. This can be a filename (relative or absolute, however user specified it), but it also can be something arbitrary like "$if / $elseif condition". So don't treat it like a reliable filename, for this use StreamAbsoluteFileName. | |
| Public | property StreamAbsoluteFileName: string read FStreamAbsoluteFileName; |
|
Filename, always absolute, of the underlying file of this stream. Empty ('') if this is not a file stream. | |
Generated by PasDoc 0.17.0.snapshot.