Unit PasDoc_Tokenizer
Description
Simple Pascal tokenizer.
The TTokenizer object creates TToken objects (tokens) for the Pascal programming language from a character input stream.
The PasDoc_Scanner unit does the same (it actually uses this unit's tokenizer), with the exception that it evaluates compiler directives, which are comments that start with a dollar sign.
Source: source/component/PasDoc_Tokenizer.pas (line 38).
Uses
- Classes
- Generics.Collections
- PasDoc_Utils
- PasDoc_Types
- PasDoc_StreamUtils
Overview
Classes, Interfaces, Objects and Records
| Name | Description |
|---|---|
Class TToken |
Stores the exact type and additional information on one token. |
Class TTokenizer |
Converts an input TStream to a sequence of TToken objects. |
Functions and Procedures
function StandardDirectiveByName(const Name: string): TStandardDirective; |
function KeyWordByName(const Name: string): TKeyword; |
function IsIdentifierStartChar(const C: Char): Boolean; |
function IsIdentifierOtherChar(const C: Char): Boolean; |
Types
TTokenType = (...); |
TKeyword = (...); |
TStandardDirective = (...); |
TStandardDirectives = set of TStandardDirective; |
TSymbolType = (...); |
TTokenList = specialize TObjectList<TToken>; |
Constants
TOKEN_TYPE_NAMES: array[TTokenType] of string =
( 'whitespace', 'comment ((**)-style)', 'comment ({}-style)',
'comment (///-style)',
'comment (//-style)', 'identifier', 'number', 'string', 'symbol',
'directive', 'reserved word', 'AT&T assembler register name', 'double-quoted string'); |
TokenCommentTypes: set of TTokenType =
[ TOK_COMMENT_PAS, TOK_COMMENT_EXT,
TOK_COMMENT_HELPINSIGHT,
TOK_COMMENT_CSTYLE ]; |
SymbolNames: array[TSymbolType] of string =
( '+', '-', '*', '/', '=', '<', '<=', '>', '>=', '[', ']', ',',
'(', ')', ':', ';', 'ˆ', '.', '@', '$', ':=', '..', '**', '\', '"' ); |
KeyWordArray: array[Low(TKeyword)..High(TKeyword)] of string =
('x',
'AND', 'ARRAY', 'AS', 'ASM', 'BEGIN', 'CASE', 'CLASS', 'OBJCCLASS', 'CONST',
'CONSTRUCTOR', 'DESTRUCTOR', 'DISPINTERFACE', 'DIV', 'DO', 'DOWNTO',
'ELSE', 'END', 'EXCEPT', 'EXPORTS', 'FILE', 'FINALIZATION',
'FINALLY', 'FOR', 'FUNCTION', 'GOTO', 'IF', 'IMPLEMENTATION',
'IN', 'INHERITED', 'INITIALIZATION', 'INLINE', 'INTERFACE',
'IS', 'LABEL', 'LIBRARY', 'MOD', 'NIL', 'NOT', 'OBJECT', 'OF',
'OR', 'PACKED', 'PROCEDURE', 'PROGRAM', 'PROPERTY',
'RAISE', 'RECORD', 'REPEAT', 'RESOURCESTRING', 'SET', 'SHL',
'SHR', 'STRING', 'THEN', 'THREADVAR', 'TO', 'TRY', 'TYPE',
'UNIT', 'UNTIL', 'USES', 'VAR', 'WHILE', 'WITH', 'XOR'); |
StandardDirectiveArray:
array[Low(TStandardDirective)..High(TStandardDirective)] of PChar =
('x',
'ABSOLUTE', 'ABSTRACT', 'APIENTRY', 'ASSEMBLER', 'AUTOMATED',
'CDECL', 'CVAR', 'DEFAULT', 'DISPID', 'DYNAMIC', 'EXPERIMENTAL', 'EXPORT', 'EXTERNAL',
'FAR', 'FORWARD', 'GENERIC', 'HELPER', 'INDEX', 'INLINE', 'MESSAGE', 'NAME', 'NEAR',
'NODEFAULT', 'NORETURN', 'ON', 'OPERATOR', 'OUT', 'OVERLOAD', 'OVERRIDE', 'PASCAL', 'PRIVATE',
'PROTECTED', 'PUBLIC', 'PUBLISHED', 'READ', 'REFERENCE', 'REGISTER',
'REINTRODUCE', 'RESIDENT', 'SEALED', 'SPECIALIZE', 'STATIC',
'STDCALL', 'STORED', 'STRICT', 'VIRTUAL',
'WRITE', 'DEPRECATED', 'SAFECALL', 'PLATFORM', 'VARARGS', 'FINAL',
'UNIMPLEMENTED'); |
Description
Functions and Procedures
function StandardDirectiveByName(const Name: string): TStandardDirective; |
|
Checks is Name (case ignored) some Pascal keyword. Returns SD_INVALIDSTANDARDDIRECTIVE if not. |
function KeyWordByName(const Name: string): TKeyword; |
|
Checks is Name (case ignored) some Pascal standard directive. Returns KEY_INVALIDKEYWORD if not. |
function IsIdentifierStartChar(const C: Char): Boolean; |
|
Returns true if c is a valid first character of a Pascal identifier: underscore, ASCII letter, or non-ASCII character (Unicode letter). Following https://docwiki.embarcadero.com/RADStudio/Athens/en/Identifiers , Delphi allows Unicode letters as identifier start characters. |
function IsIdentifierOtherChar(const C: Char): Boolean; |
|
Returns true if c is a valid continuation character of a Pascal identifier: underscore, ASCII letter, ASCII digit, or non-ASCII character (Unicode letter/digit). |
Types
TTokenType = (...); |
|
All types of tokens. Note that tokenizer is not able to tell whether you used standard directive (e.g. 'Register') as an identifier (e.g. you're declaring procedure named 'Register') or as a real standard directive (e.g. a calling specifier 'register'). So there is no token like TOK_STANDARD_DIRECTIVE here, standard directives are always reported as TOK_IDENTIFIER. You can check TToken.Info.StandardDirective to know whether this identifier is maybe used as real standard directive. Values
|
TKeyword = (...); |
|
This item has no description. Values
|
TStandardDirective = (...); |
|
This item has no description. Values
|
TStandardDirectives = set of TStandardDirective; |
|
This item has no description. |
TSymbolType = (...); |
|
enumeration type that provides all types of symbols; each symbol's name starts with SYM_ Values
|
TTokenList = specialize TObjectList<TToken>; |
|
This item has no description. |
Constants
TOKEN_TYPE_NAMES: array[TTokenType] of string =
( 'whitespace', 'comment ((**)-style)', 'comment ({}-style)',
'comment (///-style)',
'comment (//-style)', 'identifier', 'number', 'string', 'symbol',
'directive', 'reserved word', 'AT&T assembler register name', 'double-quoted string'); |
|
Names of the token types. All start with lower letter. They should somehow describe (in a few short words) given TTokenType. |
TokenCommentTypes: set of TTokenType =
[ TOK_COMMENT_PAS, TOK_COMMENT_EXT,
TOK_COMMENT_HELPINSIGHT,
TOK_COMMENT_CSTYLE ]; |
|
This item has no description. |
SymbolNames: array[TSymbolType] of string =
( '+', '-', '*', '/', '=', '<', '<=', '>', '>=', '[', ']', ',',
'(', ')', ':', ';', 'ˆ', '.', '@', '$', ':=', '..', '**', '\', '"' ); |
|
Symbols as strings. They can be useful to have some mapping TSymbolType -> string, but remember that actually some symbols in tokenizer have multiple possible representations, e.g. "right bracket" is usually given as "]" but can also be written as ".)". |
KeyWordArray: array[Low(TKeyword)..High(TKeyword)] of string =
('x',
'AND', 'ARRAY', 'AS', 'ASM', 'BEGIN', 'CASE', 'CLASS', 'OBJCCLASS', 'CONST',
'CONSTRUCTOR', 'DESTRUCTOR', 'DISPINTERFACE', 'DIV', 'DO', 'DOWNTO',
'ELSE', 'END', 'EXCEPT', 'EXPORTS', 'FILE', 'FINALIZATION',
'FINALLY', 'FOR', 'FUNCTION', 'GOTO', 'IF', 'IMPLEMENTATION',
'IN', 'INHERITED', 'INITIALIZATION', 'INLINE', 'INTERFACE',
'IS', 'LABEL', 'LIBRARY', 'MOD', 'NIL', 'NOT', 'OBJECT', 'OF',
'OR', 'PACKED', 'PROCEDURE', 'PROGRAM', 'PROPERTY',
'RAISE', 'RECORD', 'REPEAT', 'RESOURCESTRING', 'SET', 'SHL',
'SHR', 'STRING', 'THEN', 'THREADVAR', 'TO', 'TRY', 'TYPE',
'UNIT', 'UNTIL', 'USES', 'VAR', 'WHILE', 'WITH', 'XOR'); |
|
all Object Pascal keywords |
StandardDirectiveArray:
array[Low(TStandardDirective)..High(TStandardDirective)] of PChar =
('x',
'ABSOLUTE', 'ABSTRACT', 'APIENTRY', 'ASSEMBLER', 'AUTOMATED',
'CDECL', 'CVAR', 'DEFAULT', 'DISPID', 'DYNAMIC', 'EXPERIMENTAL', 'EXPORT', 'EXTERNAL',
'FAR', 'FORWARD', 'GENERIC', 'HELPER', 'INDEX', 'INLINE', 'MESSAGE', 'NAME', 'NEAR',
'NODEFAULT', 'NORETURN', 'ON', 'OPERATOR', 'OUT', 'OVERLOAD', 'OVERRIDE', 'PASCAL', 'PRIVATE',
'PROTECTED', 'PUBLIC', 'PUBLISHED', 'READ', 'REFERENCE', 'REGISTER',
'REINTRODUCE', 'RESIDENT', 'SEALED', 'SPECIALIZE', 'STATIC',
'STDCALL', 'STORED', 'STRICT', 'VIRTUAL',
'WRITE', 'DEPRECATED', 'SAFECALL', 'PLATFORM', 'VARARGS', 'FINAL',
'UNIMPLEMENTED'); |
|
Object Pascal directives |
Authors
Generated by PasDoc 1.0.2.