PasDoc's autodoc

Class TTokenizer

Description
Hierarchy
Fields
Methods
Properties

Unit

PasDoc_Tokenizer

Declaration

type TTokenizer = class(TObject)

Description

Converts an input TStream to a sequence of TToken objects.

Hierarchy

Overview

Fields

Protected FOnMessage: TPasDocMessageEvent;
Protected FVerbosity: Cardinal;
Protected BufferedChar: Char;
Protected EOS: Boolean;
Protected IsCharBuffered: Boolean;
Protected Row: Integer;
Protected Stream: TStream;
Protected FStreamName: string;
Protected FStreamPath: string;

Methods

Protected procedure DoError(const AMessage: string; const AArguments: array of const);
Protected procedure DoMessage(const AVerbosity: Cardinal; const MessageType: TPasDocMessageType; const AMessage: string; const AArguments: array of const);
Protected procedure CheckForDirective(const t: TToken);
Protected procedure ConsumeChar;
Protected function CreateSymbolToken(const st: TSymbolType; const s: string): TToken; overload;
Protected function CreateSymbolToken(const st: TSymbolType): TToken; overload;
Protected function GetChar(out c: AnsiChar): Integer;
Protected function PeekChar(out c: Char): Boolean;
Protected function ReadCommentType1: TToken;
Protected function ReadCommentType2: TToken;
Protected function ReadCommentType3: TToken;
Protected function ReadAttAssemblerRegister: TToken;
Protected function ReadLiteralString(var t: TToken): Boolean;
Protected function ReadToken(c: Char; const s: TCharSet; const TT: TTokenType; var t: TToken): Boolean;
Public constructor Create( const AStream: TStream; const OnMessageEvent: TPasDocMessageEvent; const VerbosityLevel: Cardinal; const AStreamName, AStreamPath: string);
Public destructor Destroy; override;
Public function HasData: Boolean;
Public function GetStreamInfo: string;
Public function GetToken(const NilOnEnd: Boolean = false): TToken;
Public procedure UnGetToken(var T: TToken);
Public function SkipUntilCompilerDirective: TToken;

Properties

Public property OnMessage: TPasDocMessageEvent read FOnMessage write FOnMessage;
Public property Verbosity: Cardinal read FVerbosity write FVerbosity;
Public property StreamName: string read FStreamName;
Public property StreamPath: string read FStreamPath;

Description

Fields

Protected FOnMessage: TPasDocMessageEvent;
 
Protected FVerbosity: Cardinal;
 
Protected BufferedChar: Char;

if IsCharBuffered is true, this field contains the buffered character

Protected EOS: Boolean;

true if end of stream Stream has been reached, false otherwise

Protected IsCharBuffered: Boolean;

if this is true, BufferedChar contains a buffered character; the next call to GetChar or PeekChar will return this character, not the next in the associated stream Stream

Protected Row: Integer;

current row in stream Stream; useful when giving error messages

Protected Stream: TStream;

the input stream this tokenizer is working on

Protected FStreamName: string;
 
Protected FStreamPath: string;
 

Methods

Protected procedure DoError(const AMessage: string; const AArguments: array of const);
 
Protected procedure DoMessage(const AVerbosity: Cardinal; const MessageType: TPasDocMessageType; const AMessage: string; const AArguments: array of const);
 
Protected procedure CheckForDirective(const t: TToken);
 
Protected procedure ConsumeChar;
 
Protected function CreateSymbolToken(const st: TSymbolType; const s: string): TToken; overload;
 
Protected function CreateSymbolToken(const st: TSymbolType): TToken; overload;

Uses default symbol representation, from SymbolNames[st]

Protected function GetChar(out c: AnsiChar): Integer;

Returns 1 on success or 0 on failure

Protected function PeekChar(out c: Char): Boolean;
 
Protected function ReadCommentType1: TToken;
 
Protected function ReadCommentType2: TToken;
 
Protected function ReadCommentType3: TToken;
 
Protected function ReadAttAssemblerRegister: TToken;
 
Protected function ReadLiteralString(var t: TToken): Boolean;
 
Protected function ReadToken(c: Char; const s: TCharSet; const TT: TTokenType; var t: TToken): Boolean;
 
Public constructor Create( const AStream: TStream; const OnMessageEvent: TPasDocMessageEvent; const VerbosityLevel: Cardinal; const AStreamName, AStreamPath: string);

Creates a TTokenizer and associates it with given input TStream. Note that AStream will be freed when this object will be freed.

Public destructor Destroy; override;

Releases all dynamically allocated memory.

Public function HasData: Boolean;
 
Public function GetStreamInfo: string;
 
Public function GetToken(const NilOnEnd: Boolean = false): TToken;
 
Public procedure UnGetToken(var T: TToken);

Makes the token T next to be returned by GetToken. Also sets T to Nil, to prevent you from freeing it accidentally.

You cannot have more than one "unget" token. If you only call UnGetToken after some GetToken, you are safe.

Public function SkipUntilCompilerDirective: TToken;

Skip all chars until it encounters some compiler directive, like $ELSE or $ENDIF. Returns either Nil or a token with MyType = TOK_DIRECTIVE.

Properties

Public property OnMessage: TPasDocMessageEvent read FOnMessage write FOnMessage;
 
Public property Verbosity: Cardinal read FVerbosity write FVerbosity;
 
Public property StreamName: string read FStreamName;
 
Public property StreamPath: string read FStreamPath;

This is the path where the underlying file of this stream is located.

It may be an absolute path or a relative path. Relative paths are always resolved vs pasdoc current directory. This way user can give relative paths in command-line when writing Pascal source filenames to parse.

In particular, this may be '' to indicate current dir.

It's always specified like it was processed by IncludeTrailingPathDelimiter, so it has trailing PathDelim included (unless it was '', in which case it remains empty).

Generated by PasDoc 0.16.0.