How to Tokenize String? Different Tricks in Trade!

I still remember my first project which I did seven years back. In that I got a task to split a string which contains ids separated by slash. I wrote a big snippet of string parser code by using pointers. But now when i see the following tricks, i feel – how childish was my first code snippet.

Some of the tricks are as follows.

1) strtok()

If you want plain api and no object oriented fanciness, then strtok is for you. Download the source from here.

#include "string.h"
...

// String to be splitted.
char String[] = "Long.Live_Visual.C++";

// Seperators.
char Seperators[] = "._";

// Start Tokenizing.
char* Token = strtok(String, Seperators);

// Loop until end.
while(Token != NULL)
{
 cout << Token << endl;

 // Tokenize the remaning string.
 Token = strtok(0, Seperators);
};

2) istringstream
C++ Streams are good option for string splitup. But it has only one drawback – one one delimiter can be specified. Download the source from here.

#include "iostream"
#include "sstream"
#include "string"
...

string String = "Long.Live.Visual.C++";
char Seperator = '.';

// Create input string stream.
istringstream StrStream(String);
string Token;

while(getline(StrStream, Token, Seperator))
{
 cout << Token << endl;
}

3) CString::Tokenize()

Wanna MFC way? Then CString::Tokenize() is for you. Download the source from here.

// String.
CString String = _T("long.live_Visual.C++");

// Token seperators.
CString Seperator = _T("._");
int Position = 0;
CString Token;

// Get first token.s
Token = String.Tokenize(Seperator, Position);

while(!Token.IsEmpty())
{
 wcout << Token.GetBuffer() << endl;
 Token.ReleaseBuffer();

 // Get next token.
 Token = String.Tokenize(Seperator, Position);
}

Do you know any other mind blowing tokenizing tricks? Then, share with us.