RECOGNITION OF TOKENS

 Consider the following grammar fragment: stmt → if expr then stmt

| if expr then stmt else stmt | ε

expr → term relop term

| term

                           term → id | num

where the terminals if , then, else, relop, id and num generate sets of strings given by the following regular definitions:

if → if 

then → then

else → else

relop → <|<=|=|<>|>|>=

id → letter(letter|digit)^*

num → digit⁺ (.digit⁺)?(E(+|-)?digit⁺)?

For this language fragment the lexical analyzer will recognize the keywords if, then, else, as well as the lexemes denoted by relop, id, and num. To simplify matters, we assume keywords are reserved; that is, they cannot be used as identifiers.

Transition diagrams

It is a diagrammatic representation to depict the action that will take place when a lexical analyzer is called by the parser to get the next token. It is used to keep track of information about the characters that are seen as the forward pointer scans the input.