ItuToTokens

Converts an ITU (i.e. a file like object and tokenises it into extended preprocessor tokens. This does not act on any preprocessing directives.

class cpip.core.ItuToTokens.ItuToTokens(theFileObj=None, theFileId=None, theDiagnostic=None)

Tokensises a file like object.

genTokensKeywordPpDirective()

Process the file and generate tokens. This changes the type to a keyword or preprocessing-directive if it can do so.