How to use Lexical Analysis

Introduction

A TLex object may be constructed as an empty object, from another TLex, or from an existing string as below. The code fragments are taken from an example that implements a simple Reverse Polish Notation calculation engine.

TInt RPNCalc(const TDesC& aCommand, TReal& aReturnValue)
    {
    ...
    TLex input (aCommand) ;
    ...

Code can then proceed to move through the TLex data to:

• mark positions for rolling back to later

• delimit the start of lexical tokens

• delete parts of the string held by the TLex object.

...
input.Mark() ;                     // Remember where we are.
input.SkipCharacters() ;           // Move to end of character token.
...
_LIT(KTextMemset,"MEMSET");
if ( input.TokenLength() != 0 )    // if valid potential token
    {
    TPtrC token = input.MarkedToken() ;    // then extract token
    if ( token.CompareF(KTextMemset == 0)  // and test.
        {
        ...
        }
    ...
    }
 ...

Analysis can also be done by character using functions that move through the TLex data, extracting, returning and jumping specified character lengths. For example:

...
// ensure we are looking at a digit or sign
if (!(input.Peek().IsDigit() || (input.Peek() == '.') ) )
    {
    return KErrNotFound ;
    }
...    
// deal with sign
if (input.Peek() == '+')
    {
    input.Inc();
    }

Additionally, numeric conversion functions permit a variety of numeric formats to be extracted from the TLex data, with provision for conversion using the most common number systems (radixes).

...
if (input.Val(extractUint) == KErrNone)
    {
    stack.Push(TReal(extractUint));
    }
else if (input.Val(extractReal) == KErrNone)
        {
        stack.Push(extractReal);
        }
...

where stack, is an instance of a class implementing a stack.

Constructing TLex objects

This converts an real number into a string, which is then assigned to a TLex.

TBuf<0x100> convertRealToString;
// want a TLex from a value
if (convertRealToString.Num(value,format) < KErrNone )
    {
       ...
       }
else
    {
    convertRealToString.ZeroTerminate();
    TLex string(convertRealToString) ;
       }

This takes a descriptor as a function parameter, and copies it to a TLex.

TInt RPNCalc(const TDesC& iCommand, TReal& returnValue)
    {
    TLex input (iCommand) ;
       }

Peeking the next character

This shows a code flow decision made according to next character to be read from the TLex:

if (!(input.Peek()).IsDigit()) // found non-digit after decimal point if 

Moving past a character that has been peeked

This shows the use of the Inc() function to move past a character that has been peeked:

if (input.Peek() == '-') 
    {
    input.Inc() ;        // move past minus sign & flag
    negative = ETrue ;
    }

Restoring a previously “got” character

This shows the use of UnGet() to restore the previously "got" character.

....
if (input.Offset() > 0)    // if not at start of line
    {
    input.UnGet() ;        // restore 'got' character
    }

If the previous character is before the start of the string, then the function raises a USER 59 panic for the TLex8 variant and a USER 64 panic for the TLex16 variant.

Reset the next character to the supplied mark

This shows how to allow part of a TLex can be parsed again:

if (!(input.Peek()).IsDigit())
    {
    // found non-digit after decimal point. Error, so rewind 
    input.UnGetToMark(startMark);
    }

Skipping any non-white space to get the next token

This parses a TLex for the next token:

input.Mark() ;                      // remember where we are
input.SkipCharacters() ;            // move to end of character token
if ( input.TokenLength() != 0 )     // if valid potential token
...

Getting length of a token

This shows how TokenLength() is used to return the difference between the position of the next character and the extraction mark. This gives a check as to whether the token length is valid. An invalid token length implies an invalid token.

if ( input.TokenLength() != 0 )  // if valid token length
...

Extracting a token

This extracts a marked token.

TPtrC token = input.MarkedToken() ;  // extract token 

Getting the offset of next character position

This shows how to return the offset of the next character position from the start of the string.

if (input.Offset() > 0)    // if not at start of line
    {
    input.UnGet() ;        // restore 'got' character
    ...
    }

Extracting an unknown number type

This example shows how to return the offset of the next character position from the start of the string.

if (input.Val(extractUint) == KErrNone)
    {
    stack.Push(TReal(extractUint)) ;
    }
else 
    {
    if (input.Val(extractReal) == KErrNone)
        {
        stack.Push(extractReal) ;
        }
    }

This extracts an unknown number type. Tries an integer first and then, if this fails, tries a real:

if (input.Val(extractUint) == KErrNone)
    stack.Push(TReal(extractUint)) ;
else if (input.Val(extractReal) == KErrNone)
    stack.Push(extractReal) ;