As someone else pointed out what you really want is the implementation of strspn/strcspn which is where the loop is. You don't "carry" that code along. You just write

    while (i < len && !is_token_begin(buf[i]))
        i++;
    if (i == len)
        error("End of input\n");
    start_token(tok);
    while (i < len && is_token_char(buf[i])) {
        i++;
        add_to_token(tok, buf[i]);
    }
    end_token(tok);

or something along those lines. Whatever you need. It's not rocket science. Putting highly fluctuating and project-specific code like this in a library would only have disadvantages. Not everything should be in a library. In fact, most things should not be.