The Lexer Interface

#  | The Lexer Iterator Interface |

> Introduction
> Wave Lexer synopsis
> Public Typedefs
> Member functions

## Introduction

Every lexer, which should be used in conjunction with the `Wave` library, has to return tokens formed from the input stream. These tokens should conform to the synopsis described in the topic The Token Type. The lexer type should expose an interface which conforms at least to a `forward_iterator` (in the sense defined by the the C++ Standard) returning the token type. The code sample below does not show the definition of this forward iterator interface because this is highly implementation defined.

## Wave lexer synopsis (header: wave/cpplexer/cpp_lexer_interface.hpp)

        struct lex_iterator
        {
            typedef boost::wave::lex_token<> token_type;

        // Every lex_iterator should implement at least a complete
        // forward_iterator interface (not shown here)
            typedef std::forward_iterator_tag iterator_category;

        // additional requirements
            lex_iterator();

            template <typename Iterator>
            lex_iterator(Iterator const &first, Iterator const &last
                typename token_type::position_type const &pos,
                boost::wave::language_support language)
        };

Please note, that the `lex_iterator` defined in the library header wave/cpplexer/cpp_lexer_interface.hpp actually is a template class taking the token type to use as its template parameter. This is omitted in the synopsis above because it is an implementation detail of the Re2C lexer provided as part of the Wave library.

If you want to use Wave in conjunction with your own lexing component this will have to conform to the interface described above only.

## Public Typedefs

Besides the typedefs mandated for a `forward_iterator` by the C++ standard every lexer to be used with the `Wave` library should define the following typedefs:

Public typedef's defined by the `boost::wave::context` class
`token_type` | The `token` type returned by the lexer. This is type is used as the return value of the main iterators provided by the `boost::wave::context` object too.

## Member functions

Besides the functions, which should be provided for `forward_iterators` as mandated by the C++ Standard, every lexer must implement the following functions to be used with the `Wave` library:

constructor

        lex_iterator();

> The default constructor should construct a lexer iterator, which may be used as the end iterator of the provided iterator range.

        template <typename Iterator>
        lex_iterator(Iterator const &first, Iterator const &last,
            typename token_type::position_type const &pos,
            boost::wave::language_support language);

> The second constructor should construct a lexer iterator, which may be used as a iterator traversing over the token sequence, generated by the lexer class.

> The pair of iterators `first` and `last` should represent the input stream to be tokenized by the given lexer class.

> The parameter `pos` contains the initial position information to be used for token generation.

> The parameter `language` controls the reuqired mode with which the lexer should be initialised.

Copyright © 2003-2005 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

Last updated: Thursday, December 1, 2005 21:25