PpLexer

Generates tokens from a C or C++ translation unit.

TODO: Fix accidental token pasting.

TODO: Preprocessor statements in arguments of function like macros. Sect. 3.9 of cpp.pdf and existing MacroEnv tests.

exception cpip.core.PpLexer.ExceptionPpLexer

Exception when handling PpLexer object.

exception cpip.core.PpLexer.ExceptionPpLexerAlreadyGenerating

Exception when two generators are created then the internal state will become inconsistent.

exception cpip.core.PpLexer.ExceptionPpLexerCallStack

Exception when finding issues with the call stack or nested includes.

exception cpip.core.PpLexer.ExceptionPpLexerCallStackTooSmall

Exception when sys.getrecursionlimit() is too small.

exception cpip.core.PpLexer.ExceptionPpLexerCondLevelOutOfRange

Exception when handling a conditional token generation level.

exception cpip.core.PpLexer.ExceptionPpLexerDefine

Exception when loading predefined macro definitions.

exception cpip.core.PpLexer.ExceptionPpLexerNestedInclueLimit

Exception when nested #include limit exceeded.

exception cpip.core.PpLexer.ExceptionPpLexerNoFile

Exception when can not find file.

exception cpip.core.PpLexer.ExceptionPpLexerPreInclude

Exception when loading pre-include files.

exception cpip.core.PpLexer.ExceptionPpLexerPreIncludeIncNoCp

Exception when loading a pre-include file that has no current place (e.g. a StringIO object) and the pre-include then has an #include statement.

exception cpip.core.PpLexer.ExceptionPpLexerPredefine

Exception when loading predefined macro definitions.

cpip.core.PpLexer.PREPROCESSING_DIRECTIVES = ['if', 'ifdef', 'ifndef', 'elif', 'else', 'endif', 'include', 'define', 'undef', 'line', 'error', 'pragma']

Allowable preprocessing directives

class cpip.core.PpLexer.PpLexer(tuFileId, includeHandler, preIncFiles=None, diagnostic=None, pragmaHandler=None, stdPredefMacros=None, autoDefineDateTime=True, gccExtensions=False)

Create a translation unit tokeniser that applies ISO/IEC 9899:1999(E) Section 6 and ISO/IEC 14882:1998(E) section 16.

tuFileId

A file ID that will be given to the include handler to find the translation unit. Typically this will be the file path (as a string) to the file that is the Initial Translation Unit (ITU) i.e. the file being preprocessed.

includeHandler

A handler to file #include‘d files typically a IncludeHandler.IncludeHandlerStd(). This might have user and system include path information and a means of resolving file references.

preIncFiles

An ordered list of file like objects that are pre-include files. These are processed in order before the ITU is processed. Macro redefinition rules apply.

diagnostic

A diagnostic object, defaults to a CppDiagnostic.PreprocessDiagnosticStd().

pragmaHandler

A handler for #pragma statements. This shall have a function pragma() defined that takes a non-zero length list of PpTokens the last of which will be a newline token.

stdPredefMacros

A dictionary of Standard pre-defined macros. See for example: ISO/IEC 9899:1999 (E) 6.10.8 “Predefined macro names”, ISO/IEC 14882:1998 (E) 16.8 “Predefined macro names” and N2800=08-0310 16.8 “Predefined macro names”. The macros __DATE__ and __TIME__ will be automatically updated to current locale date/time (see autoDefineDateTime).

autoDefineDateTime

If True then the macros __DATE__ and __TIME__ will be automatically updated to current locale date/time. Mostly this is used for testing.

gccExtensions

Support GCC extensions. Currently just #include_next is supported.

TODO: Set flags here rather than supplying them to a generator? This would make the API simply the ctor and ppTokens/next(). Flags would be:

incWs - Include whitespace tokens.

condLevel - (0, 1, 2) thus:

0. No conditionally compiled tokens. The fileIncludeGraphRoot will

   not have any information about conditionally included files.

1. Conditionally compiled tokens are generated but not from

   conditionally included files. The fileIncludeGraphRoot will have a reference to a conditionally included file but not that included file’s includes.

2. Conditionally compiled tokens including tokens from conditionally

   included files. The fileIncludeGraphRoot will have all the information about conditionally included files recursively.

CALL_STACK_DEPTH_ASSUMED_PPTOKENS = 10

Each include The call stack depth, D = A + B + C*L Where L is the number of levels of nested includes and A is the call stack A above:

CALL_STACK_DEPTH_FIRST_INCLUDE = 3

B above:

CALL_STACK_DEPTH_PER_INCLUDE = 3

C above:

COND_LEVEL_DEFAULT = 0

Conditianlity settings for token generation

COND_LEVEL_OPTIONS = [0, 1, 2]

Conditionality level (0, 1, 2)

MAX_INCLUDE_DEPTH = 200

The maximum value of nested #include’s

colNum

Returns the current column number as an integer during processing.

condCompGraph

The conditional compilation graph as a CppCond.CppCondGraph object.

condState

The conditional state as (boolean, string).

currentFile

Returns the file ID on the top of the file stack.

definedMacros

Returns a string representing the currently defined macros.

fileIncludeGraphRoot

Returns the FileIncludeGraphRoot object.

fileLineCol

Returns a FileLineCol object or None

fileName

Returns the current file name during processing.

fileStack

Returns the file stack.

finalise()

Finalisation, may raise any Exception.

includeDepth

Returns the integer depth of the include stack.

lineNum

Returns the current line number as an integer during processing or None.

macroEnvironment

The current Macro environment as a MacroEnv object. Caution: Write to this at your own risk. Your write might be ignored or cause undefined behaviour.

ppTokens(incWs=True, minWs=False, condLevel=0)

A generator for providing PpToken.PpTokens to section 16 of ISO/IEC 14882:1998(E).

incWs - if True than whitespace tokens are included (i.e. tok.isWs() == True).

minWs - if True then whitespace runs will be minimised to a single space or, if newline is in the whitespce run, a single newline

condLevel - if True then conditionally compiled tokens will be yielded and they will have have tok.isCond == True. The fileIncludeGraphRoot will be marked up with the appropriate conditionality. Levels are:

0: No conditionally compiled tokens. The fileIncludeGraphRoot will
not have any information about conditionally included files.

1: Conditionally compiled tokens are generated but not from 
conditionally included files. The fileIncludeGraphRoot will have
a reference to a conditionally included file but not that
included file's includes.

2: Conditionally compiled tokens including tokens from conditionally
included files. The fileIncludeGraphRoot will have all the
information about conditionally included files recursively.

(see _cppInclude where we check if self._condStack.isTrue():).

tuFileId

Returns the user supplied ID of the translation unit.

cpip.core.PpLexer.UNNAMED_FILE_NAME = 'Unnamed Pre-include'

Used when file objects have no name