123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290 |
- <html>
- <head>
- <title>In-depth The Scanner</title>
- <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
- <link rel="stylesheet" href="theme/style.css" type="text/css">
- </head>
- <body>
- <table width="100%" border="0" background="theme/bkd2.gif" cellspacing="2">
- <tr>
- <td width="10">
- </td>
- <td width="85%"> <font size="6" face="Verdana, Arial, Helvetica, sans-serif"><b>In-depth:
- The Scanner</b></font> </td>
- <td width="112"><a href="http://spirit.sf.net"><img src="theme/spirit.gif" width="112" height="48" align="right" border="0"></a></td>
- </tr>
- </table>
- <br>
- <table border="0">
- <tr>
- <td width="10"></td>
- <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td>
- <td width="30"><a href="indepth_the_parser.html"><img src="theme/l_arr.gif" border="0"></a></td>
- <td width="30"><a href="indepth_the_parser_context.html"><img src="theme/r_arr.gif" border="0"></a></td>
- </tr>
- </table>
- <h2>Basic Scanner API </h2>
- <table width="90%" border="0" align="center">
- <tr>
- <td class="table_title" colspan="10"> class scanner </td>
- </tr>
- <tr>
- <tr>
- <td class="table_cells"><code><span class=identifier>value_t</span></code></td>
- <td class="table_cells">typedef: The value type of the scanner's iterator</td>
- </tr>
- <td class="table_cells"><code><span class=identifier>ref_t</span></code></td>
- <td class="table_cells">typedef: The reference type of the scanner's iterator</td>
- </tr>
- <td class="table_cells"><code><span class=keyword>bool </span><span class=identifier>at_end</span><span class=special>()
- </span><span class=keyword>const</span></code></td>
- <td class="table_cells">Returns true if the input is exhausted</td>
- </tr>
- <td class="table_cells"><code><span class=identifier>value_t </span><span class=keyword>operator</span><span class=special>*()
- </span><span class=keyword>const</span></code></td>
- <td class="table_cells">Dereference/get a <code><span class=identifier>value_t</span></code>
- from the input</td>
- </tr>
- <td class="table_cells"><code><span class=keyword> </span><span class=identifier>scanner
- </span><span class=keyword>const</span><span class=special>& </span><span class=keyword>operator</span><span class=special>++()</span></code></td>
- <td class="table_cells">move the scanner forward</td>
- </tr>
- <tr>
- <td class="table_cells"><code><span class=identifier>IteratorT& first</span><span class=special></span></code></td>
- <td class="table_cells">The iterator pointing to the current input position.
- Held by reference</td>
- </tr>
- <tr>
- <td class="table_cells"><code><span class=identifier>IteratorT </span><span class=keyword>const</span>
- <span class=identifier>last</span><span class=special></span></code></td>
- <td class="table_cells">The iterator pointing to the end of the input. Held
- by value</td>
- </tr>
- </table>
- <p> The basic behavior of the scanner is handled by policies. The actual execution
- of the scanner's public member functions listed in the table above is implemented
- by the scanner policies.</p>
- <p> Three sets of policies govern the behavior of the scanner. These policies
- make it possible to extend Spirit non-intrusively. The scanner policies allow
- the core-functionality to be extended without requiring any potentially destabilizing
- changes to the code. A library writer might provide her own policies that override
- the ones that are already in place to fine tune the parsing process
- to fit her own needs. Layers above the core might also want to take advantage
- of this policy based machanism. Abstract syntax tree generation, debuggers and
- lexers come to mind.</p>
- <p> There are three sets of policies that govern:</p>
- <ul>
- <li>Iteration and filtering</li>
- <li>Recognition and matching</li>
- <li>Handling semantic actions</li>
- </ul>
- <a name="iteration_policy"></a>
- <h2>iteration_policy</h2>
- <p> Here are the default policies that govern iteration and filtering:</p>
- <pre>
- <code><span class=keyword>struct </span><span class=identifier>iteration_policy
- </span><span class=special>{
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>>
- </span><span class=keyword>void
- </span><span class=identifier>advance</span><span class=special>(</span><span class=identifier>ScannerT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>scan</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{ </span><span class=special>++</span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>first</span><span class=special>; </span><span class=special>}
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>>
- </span><span class=keyword>bool </span><span class=identifier>at_end</span><span class=special>(</span><span class=identifier>ScannerT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>scan</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{ </span><span class=keyword>return </span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>first </span><span class=special>== </span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>last</span><span class=special>; </span><span class=special>}
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>T</span><span class=special>>
- </span><span class=identifier>T </span><span class=identifier>filter</span><span class=special>(</span><span class=identifier>T </span><span class=identifier>ch</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{ </span><span class=keyword>return </span><span class=identifier>ch</span><span class=special>; </span><span class=special>}
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>>
- </span><span class=keyword>typename </span><span class=identifier>ScannerT</span><span class=special>::</span><span class=identifier>ref_t
- </span><span class=identifier>get</span><span class=special>(</span><span class=identifier>ScannerT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>scan</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{ </span><span class=keyword>return </span><span class=special>*</span><span class=identifier>scan</span><span class=special>.</span><span class=identifier>first</span><span class=special>; </span><span class=special>}
- </span><span class=special>};</span></code></pre>
- <table width="90%" border="0" align="center">
- <tr>
- <td class="table_title" colspan="8"> Iteration and filtering policies </td>
- </tr>
- <tr>
- <tr>
- <td class="table_cells"><b>advance</b></td>
- <td class="table_cells">Move the iterator forward</td>
- </tr>
- <td class="table_cells"><b>at_end</b></td>
- <td class="table_cells">Return true if the input is exhausted</td>
- </tr>
- <td class="table_cells"><b>filter</b></td>
- <td class="table_cells">Filter a character read from the input</td>
- </tr>
- <td class="table_cells"><b>get</b></td>
- <td class="table_cells">Read a character from the input</td>
- </tr>
- </table>
- <p> The following code snippet demonstrates a simple policy that converts all
- characters to lower case:</p>
- <pre>
- <code><span class=keyword>struct </span><span class=identifier>inhibit_case_iteration_policy </span><span class=special>: </span><span class=keyword>public </span><span class=identifier>iteration_policy
- </span><span class=special>{
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>CharT</span><span class=special>>
- </span><span class=identifier>CharT filter</span><span class=special>(</span><span class=identifier>CharT ch</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{
- </span><span class=keyword>return </span>std::<span class=identifier>tolower</span><span class=special>(</span><span class=identifier>ch</span><span class=special>);
- }
- };</span></code></pre>
- <a name="match_policy"></a>
- <h2>match_policy</h2>
- <p> Here are the default policies that govern recognition and matching:</p>
- <pre>
- <code><span class=keyword>struct </span><span class=identifier>match_policy
- </span><span class=special>{
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>T</span><span class=special>>
- </span><span class=keyword>struct </span><span class=identifier>result </span><span class=special>
- {
- </span><span class=keyword>typedef </span><span class=identifier>match</span><span class=special><</span><span class=identifier>T</span><span class=special>> </span><span class=identifier>type</span><span class=special>; </span><span class=special>
- };
- </span><span class=keyword>const </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>>
- </span><span class=identifier>no_match</span><span class=special>() </span><span class=keyword>const
- </span><span class=special>{ </span><span class=keyword>
- return </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>>(); </span><span class=special>
- }
- </span><span class=keyword>const </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>>
- </span><span class=identifier>empty_match</span><span class=special>() </span><span class=keyword>const
- </span><span class=special>{ </span><span class=keyword>
- return </span><span class=identifier>match</span><span class=special><</span><span class=identifier>nil_t</span><span class=special>>(</span><span class=number>0</span><span class=special>, </span><span class=identifier>nil_t</span><span class=special>());
- </span><span class=special>}
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>AttrT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>IteratorT</span><span class=special>>
- </span><span class=identifier>match</span><span class=special><</span><span class=identifier>AttrT</span><span class=special>>
- </span><span class=identifier>create_match</span><span class=special>(
- </span><span class=keyword>std::size_t </span><span class=identifier>length</span><span class=special>,
- </span><span class=identifier>AttrT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>val</span><span class=special>,
- </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*first*/</span><span class=special>,
- </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*last*/</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{ </span><span class=keyword>
- return </span><span class=identifier>match</span><span class=special><</span><span class=identifier>AttrT</span><span class=special>>(</span><span class=identifier>length</span><span class=special>, </span><span class=identifier>val</span><span class=special>); </span><span class=special>
- }
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>MatchT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>IteratorT</span><span class=special>>
- </span><span class=keyword>void
- </span><span class=identifier>group_match</span><span class=special>(
- </span><span class=identifier>MatchT</span><span class=special>& </span><span class=comment>/*m*/</span><span class=special>,
- </span><span class=identifier>parser_id </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*id*/</span><span class=special>,
- </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*first*/</span><span class=special>,
- </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=comment>/*last*/</span><span class=special>) </span><span class=keyword>const </span><span class=special>{}
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>Match1T</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>Match2T</span><span class=special>>
- </span><span class=keyword>void
- </span><span class=identifier>concat_match</span><span class=special>(</span><span class=identifier>Match1T</span><span class=special>& </span><span class=identifier>l</span><span class=special>, </span><span class=identifier>Match2T </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>r</span><span class=special>) </span><span class=keyword>const
- </span><span class=special>{ </span><span class=identifier>
- l</span><span class=special>.</span><span class=identifier>concat</span><span class=special>(</span><span class=identifier>r</span><span class=special>);
- </span><span class=special>}
- </span><span class=special>};</span></code></pre>
- <table width="90%" border="0" align="center">
- <tr>
- <td class="table_title" colspan="12"> Recognition and matching </td>
- </tr>
- <tr>
- <tr>
- <td class="table_cells"><b>result</b></td>
- <td class="table_cells">A metafunction that returns a match type given an
- attribute type (see In-depth: The Parser)</td>
- </tr>
- <td class="table_cells"><b>no_match</b></td>
- <td class="table_cells">Create a failed match</td>
- </tr>
- <td class="table_cells"><b>empty_match</b></td>
- <td class="table_cells">Create an empty match. An empty match is a successful
- epsilon match (matching length == 0)</td>
- </tr>
- <td class="table_cells"><b>create_match</b></td>
- <td class="table_cells">Create a match given the matching length, an attribute
- and the iterator pair pointing to the matching portion of the input</td>
- </tr>
- <td class="table_cells"><b>group_match</b></td>
- <td class="table_cells">For non terminals such as rules, this is called after
- a successful match has been made to allow post processing</td>
- </tr>
- <td class="table_cells"><b>concat_match</b></td>
- <td class="table_cells">Concatenate two match objects</td>
- </tr>
- </table>
- <a name="action_policy"></a>
- <h2>action_policy</h2>
- <p> The action policy has only one function for handling semantic actions:</p>
- <pre>
- <code><span class=keyword>struct </span><span class=identifier>action_policy
- </span><span class=special>{
- </span><span class=keyword>template </span><span class=special><</span><span class=keyword>typename </span><span class=identifier>ActorT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>AttrT</span><span class=special>, </span><span class=keyword>typename </span><span class=identifier>IteratorT</span><span class=special>>
- </span><span class=keyword>void
- </span><span class=identifier>do_action</span><span class=special>(
- </span><span class=identifier>ActorT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>actor</span><span class=special>,
- </span><span class=identifier>AttrT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>val</span><span class=special>,
- </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>first</span><span class=special>,
- </span><span class=identifier>IteratorT </span><span class=keyword>const</span><span class=special>& </span><span class=identifier>last</span><span class=special>) </span><span class=keyword>const</span><span class=special>;
- </span><span class=special>};</span></code></pre>
- <p> The default action policy forwards to:</p>
- <pre>
- <code><span class=identifier>actor</span><span class=special>(</span><span class=identifier>first</span><span class=special>, </span><span class=identifier>last</span><span class=special>);</span></code></pre>
- <p> If the attribute <tt>val</tt> is of type nil_t. Otherwise:</p>
- <pre>
- <code><span class=identifier>actor</span><span class=special>(</span><span class=identifier>val</span><span class=special>);</span></code></pre>
- <a name="scanner_policies_mixer"></a>
- <h3>scanner_policies mixer</h3>
- <p> The class <tt>scanner_policies</tt> combines the three scanner policy classes
- above into one:</p>
- <pre>
- <code><span class=keyword>template </span><span class=special><
- </span><span class=keyword>typename </span><span class=identifier>IterationPolicyT </span><span class=special>= </span><span class=identifier>iteration_policy</span><span class=special>,
- </span><span class=keyword>typename </span><span class=identifier>MatchPolicyT </span><span class=special>= </span><span class=identifier>match_policy</span><span class=special>,
- </span><span class=keyword>typename </span><span class=identifier>ActionPolicyT </span><span class=special>= </span><span class=identifier>action_policy</span><span class=special>>
- </span><span class=keyword>struct </span><span class=identifier>scanner_policies</span><span class=special>;
- </span></code></pre>
- <p> This <i>mixer</i> class inherits from all the three policies. This scanner_policies
- class is then used to parameterize the scanner:</p>
- <pre>
- <code><span class=keyword>template </span><span class=special><
- </span><span class=keyword>typename </span><span class=identifier>IteratorT </span><span class=special>= </span><span class=keyword>char </span><span class=keyword>const</span><span class=special>*,
- </span><span class=keyword>typename </span><span class=identifier>PoliciesT </span><span class=special>= </span><span class=identifier>scanner_policies</span><span class=special><> </span><span class=special>>
- </span><span class=keyword>class </span><span class=identifier>scanner</span><span class=special>;
- </span></code></pre>
- <p> The scanner in turn inherits from the PoliciesT.</p>
- <a name="rebinding_policies"></a>
- <h3>Rebinding Policies</h3>
- <p> The scanner can be made to rebind to a different set of policies anytime.
- It has a member function <tt>change_policies(new_policies)</tt>. Given a new
- set of policies, this member function creates a new scanner with the new set
- of policies. The result type of the <i>rebound</i> scanner can be can be obtained
- by calling the metafunction:</p>
- <pre>
- <code><span class=identifier>rebind_scanner_policies</span><span class=special><</span><span class=identifier>ScannerT</span><span class=special>, </span><span class=identifier>PoliciesT</span><span class=special>>::</span><span class=identifier>type</span></code></pre>
- <a name="rebinding_iterators"></a>
- <h3>Rebinding Iterators</h3>
- <p> The scanner can also be made to rebind to a different iterator type anytime.
- It has a member function <tt>change_iterator(first, last)</tt>. Given a new
- pair of iterator of type different from the ones held by the scanner, this member
- function creates a new scanner with the new pair of iterators. The result type
- of the <i>rebound</i> scanner can be can be obtained by calling the metafunction:</p>
- <pre>
- <code><span class=identifier>rebind_scanner_iterator</span><span class=special><</span><span class=identifier>ScannerT</span><span class=special>, </span><span class=identifier>IteratorT</span><span class=special>>::</span><span class=identifier>type</span></code></pre>
- <table border="0">
- <tr>
- <td width="10"></td>
- <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td>
- <td width="30"><a href="indepth_the_parser.html"><img src="theme/l_arr.gif" border="0"></a></td>
- <td width="30"><a href="indepth_the_parser_context.html"><img src="theme/r_arr.gif" border="0"></a></td>
- </tr>
- </table>
- <br>
- <hr size="1">
- <p class="copyright">Copyright © 1998-2003 Joel de Guzman<br>
- <br>
- <font size="2">Use, modification and distribution is subject to the Boost Software
- License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at
- http://www.boost.org/LICENSE_1_0.txt)</font></p>
- <p class="copyright"> </p>
- </body>
- </html>
|