Do you think that's possible?
Only the libs team can really say, and I'm not on that team.
I suppose I appreciate SmallString's ability to alias itself to Rust's String.
But I'd immediately agree that this adds hidden complexity and the potential for it to bite you in the worst possible moment.
Because Rust has &str as the lowest common denominator for all string types, "non-standard" strings aren't difficult to use.
That’s what we say now, but a few dozen years down the line? Old Mac OS versions (pre X) would store extra bits of information in the MSBs of a pointer. “32 bit clean” Mac OS had to be created to allow usage on computers with more than a few dozen megabytes. That was an all too common problem when multiple gigabytes of RAM became commonplace.
Granted, I highly doubt we’ll see exabyte level string lengths, but you never know.
Fun fact: x86-64 mandates pointers be in “canonical” form where all unused bits of the address are the same as the MSB. So on a processor that has 48 address lines, the upper 16 must match the 48th bit. If you try to access memory with a “non canonical” pointer, the processor will throw an exception. This is because they don’t want people using them for flag bits and having a repeat of the “unclean” software problem.
TL;DR: I agree, but never say never
The small problem is that your program will be limited in memory space. That's usually okay. If you want a vast expansion of memory space you probably need to start using different algorithms anyway.
The big problem is that your program depends on the CPU ignoring those bits, and now it can't run at all on updated hardware. As you noted, x86-64 solves this problem by forcing programs to clean up pointers before actually using them.
Now that you mention pointers, Apple is using unused bits in their values for signing and authentication purposes https://googleprojectzero.blogspot.com/2019/02/examining-poi...
Pretty neat in my opinion
NaN-boxing / nun-boxing is a common optimization, as seen in Firefox's SpiderMonkey, Safari's JSC, LuaJIT, etc. If all of your valid pointers are at the top of the address space, you can use 0x0 as the constant you either add/subtract or xor with your values to box/un-box. In other words, Nan-boxing/unboxing becomes a no-op.
It is used in some other places though. For instance, std::string implementations typically employ small-string optimization (SSO) which means that there might be a limit on string::size() since some of the bits inside that value are used for different purposes (example: https://akrzemi1.wordpress.com/2014/04/14/common-optimizatio...). The "niche" in this case is the realization that strings are unlikely to be exabytes in size, so there is no reason to always store zeros in high bits of _size.