2.2.1 Syntax of the Scanner Specification Language

The notation used here for specifying the syntax of the specification language is a variant of BNF and is defined in Appendix A.

A scanner specification is allowed anywhere as an Oz statement:

<statement>

+=

<scanner specification>

It is similar to a class definition, except that it is introduced by the keyword scanner, must be named by a variable (and not an arbitrary term), since this is used for assigning file names, and allows for additional descriptors after the usual class descriptors.

<scanner specification>	::=	scanner <variable>
		{ <class descriptor> }
		{ <method> }
		{ <scanner descriptor> }+
		end

A lexical abbreviation associates an identifier with a regular expression, which can then be referenced in subsequent lexical abbreviations or any lexical rules by enclosing the identifier in curly brackets. The regular expression is additionally parenthesized when it is expanded.

<lexical abbreviation>	::=	lex <atom> "=" <regex> end
	|	lex <variable> "=" <regex> end

The definition of a lexical rule is similar to the definition of a method. However, its head consists of a regular expression; when this is matched, the body of the lexical rule is executed (as a method).

<lexical rule>	::=	lex <regex>
		<in statement>
		end

Regular expressions may be annotated with lexical modes. Each lexical mode constitutes an independent sub-scanner: At any time a certain mode is active; in this mode only the regular expressions annotated with it will be matched. All lexical rules defined within the scope of a lexical mode are annotated with this lexical mode. A lexical mode may inherit from other lexical modes; all regular expressions in these modes are then annotated with the inheriting lexical mode as well. Lexical modes implicitly inherit from all lexical modes they are nested in. Lexical rules written at top-level are annotated with the implicitly declared mode INITIAL.

<lexical mode>	::=	mode <variable> [ from { <variable> }+ ]
		{ <mode descriptor> }
		end

<mode descriptor>	::=	<lexical rule>
	|	<lexical mode>

Syntax of Regular Expressions

Regular expressions <regex> correspond to the regular expressions used in flex [Pax95] with a few exceptions:

• Gump regular expressions are either enclosed in angle brackets or given as Oz strings.

• The angle-bracket annotation with lexical modes is not supported by Gump; use scopes of lexical modes instead. Note that several distinct lexical mode definitions may occur for the same lexical mode name as long as no inheritance cycles are created.

Due to the underlying use of flex, the names of lexical abbreviations are restricted to the syntax allowed in flex name definitions.

Ambiguities and Errors in the Rule Set

Tokenization is performed by a left-to-right scan of the input character stream. If several rules match a prefix of the input, then the rule matching the longest prefix is preferred. If several rules match the same (longest) prefix of the input, then two rules may be applied to disambiguate the match (see Section 2.2.2 on how to select the rule):

First-fit.

The rule notated first in the scanner specification is preferred. In this case, every conflict can be uniquely resolved. Two errors in the rule set are possible: holes and completely covered rules (see below).

Best-fit.

Suppose two conflicting rules are rule r₁ and rule r₂, which are annotated by sets of lexical modes S₁ and S₂ respectively. Then r₁ is preferred over r₂ if and only if the following condition holds:

S₁ is a subset of S₂ and L(r₁) is a subset of L(r₂)

where L(r) is the language generated by a regular expression r, that is, the set of strings that match r. Intuitively, this rule means that r₁ is ``more specialized than'' r<sub>2</sub>. Additionally to the errors possible in the rule set in the first-fit case, here the situation may arise that the rule set is not well-ordered w. r. t. the ``more specialized than'' relation.

The following errors in the rule set may occur:

Holes in the rule set.

For some input (in some mode), no true prefix is matched by any rule. Due to the underlying implementation using flex, this will result in the warning message

"-s option given but default rule can be matched"

If at run-time some such input is encountered, this will result in an error exception

"flex scanner jammed"

Completely covered rules.

A rule r is never matched because for every prefix in L(r) exists another rule r which is preferred over r.

Non well-orderedness.

Two rules r₁ and r₂ are in conflict in the best-fit case, but neither is r₁ more specialized than r₂ nor the other way round, and no rule or set of rules exists that covers L(r₁) intersected with L(r₂).

2.2.2 Parameters to Scanner Generation

The Gump Scanner Generator supports several configuration parameters, which may be set on a per-scanner basis via the use of macro directives.

2.2.3 The Mixin Class GumpScanner.'class'

The module GumpScanner defines the runtime support needed by Gump-generated scanners. All operations and data are encapsulated in the mixin class GumpScanner.'class' that scanners have to inherit from in order to be executable.