(Original poster here.)

Yes, returning indexes is also an excellent solution in this particular case, because it forces the programmer to keep the underlying string around. And you probably can save memory by using a pair<uint32,uint32>.

Still, as several other people have pointed out, the truly idiomatic C++ solutions here are things like sub-ranges and string views. And these all use pointers internally. I deliberately chose to use an pair of pointers in place of these idiomatic types because it was easier to explain what was happening, not because I think C++ programmers should mess around with bare pointers.

What I like about the Rust version is that I can work directly with low-level types in high-performance code, and the compiler still has my back.

If you want to force the programmer to keep string around, remove the 'const' from the function signature. The compiler will rigtly complain when you call the function with a r-value as argument and will only accept l-values. l-values and r-values are the C++ way for dealing with this kind of situation. C++ has also some means to prevent from doing weird stuff.

I would say returning regular pointers is more C++-esque than your handmade range implementation. Pointers are, after all, a kind of iterator, which is an essential C++ concept. (std::string::iterator is a handful.)

(Furthermore, there is nothing "C" or wrong with regular pointers. Their only "flaw" is that they can't manage an object, but they're still the semantical way to refer to one.)

Returning regular pointers which point to memory which is not managed by you and that you by default know could disappear at any time is not the C++ way of doing anything. Thing is, a class has a public interface for a reason, and if you as an outsider want to start playing pointer games you know from the start what you are in for.

I never implied that pointers are bad, just that this particular usage of pointers is bad, and as such should not be used as an example of why we would want something else over C++. If I ever saw a function with a signature like the one proposed I'd be extremely suspicious of what was going on.

Note that this would happen in the same way if the function returned pairs of iterators. It is a choice that depends on the documented and intended usage of a function. If you feel your tokenizer will be used on temporaries (I fail to see how that would ever be useful though) you could overload the function to also take universal references or change the return parameters so that nobody can get hurt. Otherwise you return whatever delimiters you are comfortable with, with the assumption that they will not be used in some way. All code can be broken if you actively try to, so it makes no sense to protect against everything.

> Returning regular pointers which point to memory which is not managed by you and that you by default know could disappear at any time is not the C++ way of doing anything. Thing is, a class has a public interface for a reason, and if you as an outsider want to start playing pointer games you know from the start what you are in for.

Maybe not "regular pointers", but returning iterators is very common (all throughout the STL, for one thing.) Those iterators are part of the public interface, and they provide no real guarantees over bare pointers.

I don't have a horse in this race, though. I think the "nicest" solution in abstract is for `tokenize` to be a token-generator* that takes a character generator argument. If you want to store the tokens yourself you can copy them, if you're happy to stream-process them then you can do that, and if you want a reference into the original string storage you can go fly a kite.

*: Where a "token" is probably another character generator.

> All code can be broken if you actively try to, so it makes no sense to protect against everything.

That's not Rust's philosophy. Rust is memory-safe, period.

I guess technically 'if you try to' means also abusing unsafe, so in theory he's still sorta right. Every time I see someone ask a question about transmute on IRC I shiver a little.

But yes, Rust is way, way, way better in this regard.

The difference is safe-by-default languages force you to turn off the safeties before they will let you blow your own foot off. C++ doesn't believe in safeties since (at the time the language was designed) it introduces an unacceptable delay between you wanting to pull the trigger and your target and/or feet blowing up.

> I never implied that pointers are bad, just that this particular usage of pointers is bad, and as such should not be used as an example of why we would want something else over C++. If I ever saw a function with a signature like the one proposed I'd be extremely suspicious of what was going on.

You would. I wouldn't because I don't know any better. How many others are there out there that don't know any better? If you code by yourself, fine, but if you work with a team you are gonna need a higher level of training, so you could teach people like me to catch the problem. Are you always going to be disciplined enough to do code reviews that are in depth enough to capture fishiness in the type signature?

With Rust, if it compiles and passes unit tests, the code review is already 90% done.

Compilation and unit testing will not catch suboptimal expression of logic/thought (which in my experience count for way more than 10% of code reviews).

>Their only "flaw" is that they can't manage an object, but they're still the semantical way to refer to one.)

Yes, that's their only flaw. A flaw which has caused countless bugs and will continue to do so in the future. This is why C++ provides so many utilities designed to keep you from using them in 99% of cases. But yeah, that's their only flaw.

> I would say returning regular pointers is more C++-esque

In C++98 maybe ...

I would think that (in C++14 at least), std::string_view would be more appropriate than both?

http://en.cppreference.com/w/cpp/header/experimental/string_...

Using string_view in this example doesn't solve the use-after-free bug.

If you are concerned about memory safety, the only appropriate choices are to copy the string and have the string_view objects share ownership, or to use the primitive index range idea and punt the problem back to the programmer.

Yep, definitely, I was just responding to the question of which view-into-a-string type is the most appropriate.

If you really wanted the pointers, you could also just make your own class that contains the pointer vector and a copy of the entire string which would be slightly less costly than new strings for each token.

This would have advantage in case you only needed a few results; but in case you needed most of them, then substr would practically result in duplicating the memory anyway. Plus, this is sure more cumbersome than the Rust version.