The big problem with tok isn't that it alters the string; it's that it keeps state, and so can't be used in multicontext code. sep changes the interface to delegate state tracking to the caller, which is the way it should be.
Just as map and reduce in Python is "unpythonic", interfaces that mint new temp strings to process what should be a simple charstar are un-c-like, and should be avoided.
Just as map and reduce in Python is "unpythonic", interfaces that mint new temp strings to process what should be a simple charstar are un-c-like, and should be avoided.
That's not what I'm doing at all, though. I'm just giving char *'s and size_t's for portions of the original character array, not duplicating subsections of the array at all.
You are, in other words, reinventing strspn?
strspn takes a set of characters to match against and tells you how long to go until you reach a non-matching character. My tokenizer, like strtok, takes a set of delimiter characters. So same thing except with complement sets, I guess.
It's also part of a larger library that consistently uses pointer/length pairs to designate strings as opposed to null termination, which admittedly is quite non-C. No clue whether it's actually a good idea.
EDIT: If anyone's actually curious, I have it on github: http://github.com/philwelch/string
This post actually gave me an impetus to go back and add tokenization to my string library that's built around this idea.