Maybe not "regular pointers", but returning iterators is very common (all throughout the STL, for one thing.) Those iterators are part of the public interface, and they provide no real guarantees over bare pointers.
I don't have a horse in this race, though. I think the "nicest" solution in abstract is for `tokenize` to be a token-generator* that takes a character generator argument. If you want to store the tokens yourself you can copy them, if you're happy to stream-process them then you can do that, and if you want a reference into the original string storage you can go fly a kite.
*: Where a "token" is probably another character generator.
That's not Rust's philosophy. Rust is memory-safe, period.
But yes, Rust is way, way, way better in this regard.
You would. I wouldn't because I don't know any better. How many others are there out there that don't know any better? If you code by yourself, fine, but if you work with a team you are gonna need a higher level of training, so you could teach people like me to catch the problem. Are you always going to be disciplined enough to do code reviews that are in depth enough to capture fishiness in the type signature?
With Rust, if it compiles and passes unit tests, the code review is already 90% done.
I never implied that pointers are bad, just that this particular usage of pointers is bad, and as such should not be used as an example of why we would want something else over C++. If I ever saw a function with a signature like the one proposed I'd be extremely suspicious of what was going on.
Note that this would happen in the same way if the function returned pairs of iterators. It is a choice that depends on the documented and intended usage of a function. If you feel your tokenizer will be used on temporaries (I fail to see how that would ever be useful though) you could overload the function to also take universal references or change the return parameters so that nobody can get hurt. Otherwise you return whatever delimiters you are comfortable with, with the assumption that they will not be used in some way. All code can be broken if you actively try to, so it makes no sense to protect against everything.