A complementary idea is to reuse the vector of strings across different calls to `tokenize`. The main loop in the code is of the form:

    {
        std::vector<std::string> tokens = tokenize(line);
        // do work with tokens
    }

which you could change to

    std::vector<std::string> tokens;
    {
        tokens.clear();
        tokenize(line, &tokens);
        // do work with tokens
    }

Combined with your suggestion to use std::string_view, this would mean only O(1) allocations across the program for this part of the code.