123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137 |
- [/
- / Copyright (c) 2008 Eric Niebler
- /
- / Distributed under the Boost Software License, Version 1.0. (See accompanying
- / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
- /]
- [section String Splitting and Tokenization]
- _regex_token_iterator_ is the Ginsu knife of the text manipulation world. It slices! It dices! This section describes
- how to use the highly-configurable _regex_token_iterator_ to chop up input sequences.
- [h2 Overview]
- You initialize a _regex_token_iterator_ with an input sequence, a regex, and some optional configuration parameters.
- The _regex_token_iterator_ will use _regex_search_ to find the first place in the sequence that the regex matches. When
- dereferenced, the _regex_token_iterator_ returns a ['token] in the form of a `std::basic_string<>`. Which string it returns
- depends on the configuration parameters. By default it returns a string corresponding to the full match, but it could also
- return a string corresponding to a particular marked sub-expression, or even the part of the sequence that ['didn't] match.
- When you increment the _regex_token_iterator_, it will move to the next token. Which token is next depends on the configuration
- parameters. It could simply be a different marked sub-expression in the current match, or it could be part or all of the
- next match. Or it could be the part that ['didn't] match.
- As you can see, _regex_token_iterator_ can do a lot. That makes it hard to describe, but some examples should make it clear.
- [h2 Example 1: Simple Tokenization]
- This example uses _regex_token_iterator_ to chop a sequence into a series of tokens consisting of words.
- std::string input("This is his face");
- sregex re = +_w; // find a word
- // iterate over all the words in the input
- sregex_token_iterator begin( input.begin(), input.end(), re ), end;
- // write all the words to std::cout
- std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
- std::copy( begin, end, out_iter );
- This program displays the following:
- [pre
- This
- is
- his
- face
- ]
- [h2 Example 2: Simple Tokenization, Reloaded]
- This example also uses _regex_token_iterator_ to chop a sequence into a series of tokens consisting of words,
- but it uses the regex as a delimiter. When we pass a `-1` as the last parameter to the _regex_token_iterator_
- constructor, it instructs the token iterator to consider as tokens those parts of the input that ['didn't]
- match the regex.
- std::string input("This is his face");
- sregex re = +_s; // find white space
- // iterate over all non-white space in the input. Note the -1 below:
- sregex_token_iterator begin( input.begin(), input.end(), re, -1 ), end;
- // write all the words to std::cout
- std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
- std::copy( begin, end, out_iter );
- This program displays the following:
- [pre
- This
- is
- his
- face
- ]
- [h2 Example 3: Simple Tokenization, Revolutions]
- This example also uses _regex_token_iterator_ to chop a sequence containing a bunch of dates into a series of
- tokens consisting of just the years. When we pass a positive integer [^['N]] as the last parameter to the
- _regex_token_iterator_ constructor, it instructs the token iterator to consider as tokens only the [^['N]]-th
- marked sub-expression of each match.
- std::string input("01/02/2003 blahblah 04/23/1999 blahblah 11/13/1981");
- sregex re = sregex::compile("(\\d{2})/(\\d{2})/(\\d{4})"); // find a date
- // iterate over all the years in the input. Note the 3 below, corresponding to the 3rd sub-expression:
- sregex_token_iterator begin( input.begin(), input.end(), re, 3 ), end;
- // write all the words to std::cout
- std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
- std::copy( begin, end, out_iter );
- This program displays the following:
- [pre
- 2003
- 1999
- 1981
- ]
- [h2 Example 4: Not-So-Simple Tokenization]
- This example is like the previous one, except that instead of tokenizing just the years, this program
- turns the days, months and years into tokens. When we pass an array of integers [^['{I,J,...}]] as the last
- parameter to the _regex_token_iterator_ constructor, it instructs the token iterator to consider as tokens the
- [^['I]]-th, [^['J]]-th, etc. marked sub-expression of each match.
- std::string input("01/02/2003 blahblah 04/23/1999 blahblah 11/13/1981");
- sregex re = sregex::compile("(\\d{2})/(\\d{2})/(\\d{4})"); // find a date
- // iterate over the days, months and years in the input
- int const sub_matches[] = { 2, 1, 3 }; // day, month, year
- sregex_token_iterator begin( input.begin(), input.end(), re, sub_matches ), end;
- // write all the words to std::cout
- std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
- std::copy( begin, end, out_iter );
- This program displays the following:
- [pre
- 02
- 01
- 2003
- 23
- 04
- 1999
- 13
- 11
- 1981
- ]
- The `sub_matches` array instructs the _regex_token_iterator_ to first take the value of the 2nd sub-match, then
- the 1st sub-match, and finally the 3rd. Incrementing the iterator again instructs it to use _regex_search_ again
- to find the next match. At that point, the process repeats -- the token iterator takes the value of the 2nd
- sub-match, then the 1st, et cetera.
- [endsect]
|