MaxMunchGen

Generic Maximal Munch generator.

exception cpip.util.MaxMunchGen.ExceptionMaxMunchGen

Exception specialisation for MaxMunchGen.

class cpip.util.MaxMunchGen.MaxMunchGen(theGen, theFnS, isExclusive=False, yieldReplacement=False)

Provides a generator that applies Maximal munch rules.

gen()

Yields a maximal munch. If yieldReplacement is False these will be pairs of (iterable, kind) where kind is from the function, any replacement will be done on the fly. If yieldReplacement is True these will be triples of (iterable, kind, repl) where kind and repl are from the function with repl being None if no replacement. No replacement will have been done.

TODO: Reconsider this design. Really yieldReplacement decides if the underlying generator buffer contains the replacement rather than whether self yields the replacement.

cpip.util.MaxMunchGen.anyToken(theGen)

A function that always reads one token. This can be used as the last registered function to ensure that the token stream is read to completion. The kind returned is None.