Unit PasDoc_Tokenizer

Description

Uses

Classes, Interfaces, Objects and Records

Functions and Procedures

Types

Constants

Variables

Description

Simple Pascal tokenizer.

The TTokenizer object creates TToken objects (tokens) for the Pascal programming language from a character input stream.

The PasDoc_Scanner unit does the same (it actually uses this unit's tokenizer), with the exception that it evaluates compiler directives, which are comments that start with a dollar sign.

Uses

Overview

Classes, Interfaces, Objects and Records

Name	Description
Class `TToken`	Stores the exact type and additional information on one token.
Class `TTokenizer`	Converts an input TStream to a sequence of TToken objects.

Functions and Procedures

function StandardDirectiveByName(const Name: string): TStandardDirective;

function KeyWordByName(const Name: string): TKeyword;

Types

TTokenType = (...);

TKeyword = (...);

TStandardDirective = (...);

TSymbolType = (...);

Constants

TOKEN_TYPE_NAMES: array[TTokenType] of string =
  ( 'whitespace', 'comment ((**)-style)', 'comment ({}-style)',
    'comment (///-style)',
    'comment (//-style)', 'identifier', 'number', 'string', 'symbol',
    'directive', 'reserved word', 'AT&T assembler register name');

TokenCommentTypes: set of TTokenType =
  [ TOK_COMMENT_PAS, TOK_COMMENT_EXT,
  TOK_COMMENT_HELPINSIGHT,
  TOK_COMMENT_CSTYLE ];

SymbolNames: array[TSymbolType] of string =
  ( '+', '-', '*', '/', '=', '<', '<=', '>', '>=', '[', ']', ',',
    '(', ')', ':', ';', 'ˆ', '.', '@', '$', ':=', '..', '**', '\' );

KeyWordArray: array[Low(TKeyword)..High(TKeyword)] of string =
  ('x',     'AND', 'ARRAY', 'AS', 'ASM', 'BEGIN', 'CASE', 'CLASS', 'CONST',
    'CONSTRUCTOR', 'DESTRUCTOR', 'DISPINTERFACE', 'DIV',  'DO', 'DOWNTO',
    'ELSE', 'END', 'EXCEPT', 'EXPORTS', 'FILE', 'FINALIZATION',
    'FINALLY', 'FOR', 'FUNCTION', 'GOTO', 'IF', 'IMPLEMENTATION',
    'IN', 'INHERITED', 'INITIALIZATION', 'INLINE', 'INTERFACE',
    'IS', 'LABEL', 'LIBRARY', 'MOD', 'NIL', 'NOT', 'OBJECT', 'OF',
    'ON', 'OR', 'PACKED', 'PROCEDURE', 'PROGRAM', 'PROPERTY',
    'RAISE', 'RECORD', 'REPEAT', 'RESOURCESTRING', 'SET', 'SHL',
    'SHR', 'STRING', 'THEN', 'THREADVAR', 'TO', 'TRY', 'TYPE',
    'UNIT', 'UNTIL', 'USES', 'VAR', 'WHILE', 'WITH', 'XOR');

StandardDirectiveArray:
    array[Low(TStandardDirective)..High(TStandardDirective)] of PChar =
  ('x',     'ABSOLUTE', 'ABSTRACT', 'APIENTRY', 'ASSEMBLER', 'AUTOMATED',
    'CDECL', 'CVAR', 'DEFAULT', 'DISPID', 'DYNAMIC', 'EXPERIMENTAL', 'EXPORT', 'EXTERNAL',
    'FAR', 'FORWARD', 'GENERIC', 'HELPER', 'INDEX', 'INLINE', 'MESSAGE', 'NAME', 'NEAR',
    'NODEFAULT', 'OPERATOR', 'OUT', 'OVERLOAD', 'OVERRIDE', 'PASCAL', 'PRIVATE',
    'PROTECTED', 'PUBLIC', 'PUBLISHED', 'READ', 'REFERENCE', 'REGISTER',
    'REINTRODUCE', 'RESIDENT', 'SEALED', 'SPECIALIZE', 'STATIC',
    'STDCALL', 'STORED', 'STRICT', 'VIRTUAL',
    'WRITE', 'DEPRECATED', 'SAFECALL', 'PLATFORM', 'VARARGS', 'FINAL');

Description

Functions and Procedures

function StandardDirectiveByName(const Name: string): TStandardDirective;

Checks is Name (case ignored) some Pascal keyword. Returns SD_INVALIDSTANDARDDIRECTIVE if not.

function KeyWordByName(const Name: string): TKeyword;

Checks is Name (case ignored) some Pascal standard directive. Returns KEY_INVALIDKEYWORD if not.

Types

TTokenType = (...);

enumeration type that provides all types of tokens; each token's name starts with TOK_.

TOK_DIRECTIVE is a compiler directive (like $ifdef, $define).

Note that tokenizer is not able to tell whether you used standard directive (e.g. 'Register') as an identifier (e.g. you're declaring procedure named 'Register') or as a real standard directive (e.g. a calling specifier 'register'). So there is no value like TOK_STANDARD_DIRECTIVE here, standard directives are always reported as TOK_IDENTIFIER. You can check TToken.Info.StandardDirective to know whether this identifier is maybe used as real standard directive.

Values

TOK_WHITESPACE
TOK_COMMENT_PAS
TOK_COMMENT_EXT
TOK_COMMENT_HELPINSIGHT
TOK_COMMENT_CSTYLE
TOK_IDENTIFIER
TOK_NUMBER
TOK_STRING
TOK_SYMBOL
TOK_DIRECTIVE
TOK_KEYWORD
TOK_ATT_ASSEMBLER_REGISTER

TKeyword = (...);

Values

KEY_INVALIDKEYWORD
KEY_AND
KEY_ARRAY
KEY_AS
KEY_ASM
KEY_BEGIN
KEY_CASE
KEY_CLASS
KEY_CONST
KEY_CONSTRUCTOR
KEY_DESTRUCTOR
KEY_DISPINTERFACE
KEY_DIV
KEY_DO
KEY_DOWNTO
KEY_ELSE
KEY_END
KEY_EXCEPT
KEY_EXPORTS
KEY_FILE
KEY_FINALIZATION
KEY_FINALLY
KEY_FOR
KEY_FUNCTION
KEY_GOTO
KEY_IF
KEY_IMPLEMENTATION
KEY_IN
KEY_INHERITED
KEY_INITIALIZATION
KEY_INLINE
KEY_INTERFACE
KEY_IS
KEY_LABEL
KEY_LIBRARY
KEY_MOD
KEY_NIL
KEY_NOT
KEY_OBJECT
KEY_OF
KEY_ON
KEY_OR
KEY_PACKED
KEY_PROCEDURE
KEY_PROGRAM
KEY_PROPERTY
KEY_RAISE
KEY_RECORD
KEY_REPEAT
KEY_RESOURCESTRING
KEY_SET
KEY_SHL
KEY_SHR
KEY_STRING
KEY_THEN
KEY_THREADVAR
KEY_TO
KEY_TRY
KEY_TYPE
KEY_UNIT
KEY_UNTIL
KEY_USES
KEY_VAR
KEY_WHILE
KEY_WITH
KEY_XOR

TStandardDirective = (...);

Values

SD_INVALIDSTANDARDDIRECTIVE
SD_ABSOLUTE
SD_ABSTRACT
SD_APIENTRY
SD_ASSEMBLER
SD_AUTOMATED
SD_CDECL
SD_CVAR
SD_DEFAULT
SD_DISPID
SD_DYNAMIC
SD_EXPERIMENTAL
SD_EXPORT
SD_EXTERNAL
SD_FAR
SD_FORWARD
SD_GENERIC
SD_HELPER
SD_INDEX
SD_INLINE
SD_MESSAGE
SD_NAME
SD_NEAR
SD_NODEFAULT
SD_OPERATOR
SD_OUT
SD_OVERLOAD
SD_OVERRIDE
SD_PASCAL
SD_PRIVATE
SD_PROTECTED
SD_PUBLIC
SD_PUBLISHED
SD_READ
SD_REFERENCE
SD_REGISTER
SD_REINTRODUCE
SD_RESIDENT
SD_SEALED
SD_SPECIALIZE
SD_STATIC
SD_STDCALL
SD_STORED
SD_STRICT
SD_VIRTUAL
SD_WRITE
SD_DEPRECATED
SD_SAFECALL
SD_PLATFORM
SD_VARARGS
SD_FINAL

TSymbolType = (...);

enumeration type that provides all types of symbols; each symbol's name starts with SYM_

Values

SYM_PLUS
SYM_MINUS
SYM_ASTERISK
SYM_SLASH
SYM_EQUAL
SYM_LESS_THAN
SYM_LESS_THAN_EQUAL
SYM_GREATER_THAN
SYM_GREATER_THAN_EQUAL
SYM_LEFT_BRACKET
SYM_RIGHT_BRACKET
SYM_COMMA
SYM_LEFT_PARENTHESIS
SYM_RIGHT_PARENTHESIS
SYM_COLON
SYM_SEMICOLON
SYM_ROOF
SYM_PERIOD
SYM_AT
SYM_DOLLAR
SYM_ASSIGN
SYM_RANGE
SYM_POWER
SYM_BACKSLASH: SYM_BACKSLASH may occur when writing char constant "ˆ\", see ../../tests/ok_caret_character.pas

Constants

TOKEN_TYPE_NAMES: array[TTokenType] of string =
  ( 'whitespace', 'comment ((**)-style)', 'comment ({}-style)',
    'comment (///-style)',
    'comment (//-style)', 'identifier', 'number', 'string', 'symbol',
    'directive', 'reserved word', 'AT&T assembler register name');

Names of the token types. All start with lower letter. They should somehow describe (in a few short words) given TTokenType.

TokenCommentTypes: set of TTokenType =
  [ TOK_COMMENT_PAS, TOK_COMMENT_EXT,
  TOK_COMMENT_HELPINSIGHT,
  TOK_COMMENT_CSTYLE ];

SymbolNames: array[TSymbolType] of string =
  ( '+', '-', '*', '/', '=', '<', '<=', '>', '>=', '[', ']', ',',
    '(', ')', ':', ';', 'ˆ', '.', '@', '$', ':=', '..', '**', '\' );

Symbols as strings. They can be useful to have some mapping TSymbolType -> string, but remember that actually some symbols in tokenizer have multiple possible representations, e.g. "right bracket" is usually given as "]" but can also be written as ".)".

KeyWordArray: array[Low(TKeyword)..High(TKeyword)] of string =
  ('x',     'AND', 'ARRAY', 'AS', 'ASM', 'BEGIN', 'CASE', 'CLASS', 'CONST',
    'CONSTRUCTOR', 'DESTRUCTOR', 'DISPINTERFACE', 'DIV',  'DO', 'DOWNTO',
    'ELSE', 'END', 'EXCEPT', 'EXPORTS', 'FILE', 'FINALIZATION',
    'FINALLY', 'FOR', 'FUNCTION', 'GOTO', 'IF', 'IMPLEMENTATION',
    'IN', 'INHERITED', 'INITIALIZATION', 'INLINE', 'INTERFACE',
    'IS', 'LABEL', 'LIBRARY', 'MOD', 'NIL', 'NOT', 'OBJECT', 'OF',
    'ON', 'OR', 'PACKED', 'PROCEDURE', 'PROGRAM', 'PROPERTY',
    'RAISE', 'RECORD', 'REPEAT', 'RESOURCESTRING', 'SET', 'SHL',
    'SHR', 'STRING', 'THEN', 'THREADVAR', 'TO', 'TRY', 'TYPE',
    'UNIT', 'UNTIL', 'USES', 'VAR', 'WHILE', 'WITH', 'XOR');

all Object Pascal keywords

StandardDirectiveArray:
    array[Low(TStandardDirective)..High(TStandardDirective)] of PChar =
  ('x',     'ABSOLUTE', 'ABSTRACT', 'APIENTRY', 'ASSEMBLER', 'AUTOMATED',
    'CDECL', 'CVAR', 'DEFAULT', 'DISPID', 'DYNAMIC', 'EXPERIMENTAL', 'EXPORT', 'EXTERNAL',
    'FAR', 'FORWARD', 'GENERIC', 'HELPER', 'INDEX', 'INLINE', 'MESSAGE', 'NAME', 'NEAR',
    'NODEFAULT', 'OPERATOR', 'OUT', 'OVERLOAD', 'OVERRIDE', 'PASCAL', 'PRIVATE',
    'PROTECTED', 'PUBLIC', 'PUBLISHED', 'READ', 'REFERENCE', 'REGISTER',
    'REINTRODUCE', 'RESIDENT', 'SEALED', 'SPECIALIZE', 'STATIC',
    'STDCALL', 'STORED', 'STRICT', 'VIRTUAL',
    'WRITE', 'DEPRECATED', 'SAFECALL', 'PLATFORM', 'VARARGS', 'FINAL');

Object Pascal directives

Authors

Johannes Berg <johannes@sipsolutions.de>
Ralf Junker (delphi@zeitungsjunge.de)
Marco Schmidt (marcoschmidt@geocities.com)
Michalis Kamburelis
Arno Garrels <first name.name@nospamgmx.de>

Generated by PasDoc 0.15.0.

PasDoc's autodoc

Unit PasDoc_Tokenizer

Description

Uses

Overview

Classes, Interfaces, Objects and Records

Functions and Procedures

Types

Constants

Description

Functions and Procedures

Types

Values

Values

Values

Values

Constants

Authors