Just having a default in the language is insanely useful. I can only appeal to experience here (with D, which has built-in arrays), but I never want to be without them again. This goes doubly for my language, where dynamic arrays are actually a bit involved due to the need for slices, refcounting and capacity tracking for the doubling strategy on append. Not something you want to reimplement everywhere.
> It's ugly and inexperienced users will write bugs using it, but it's also tremendously useful. You mentioned quasiquoting as an alternative, but I'm not positive that it works as a preprocessor replacement for a language that lacks the "homomiconity" of LISP. Are there examples that show it works?
As an example, something like
#define SQUARE(X) ({ typeof(X) x = X; x * x; })
macro SQUARE(X) ({ typeof($X) x = $X; x * x; });
As for macros like `#define BEGIN {`, I consider it an advantage that they don't work. :)
Not something I want everywhere, in the first place. Especially refcounting.
> macro SQUARE(X) ({ typeof($X) x = $X; x * x; });
Yes, stuff like that works, but only if your replacement body is a fully formed syntactical expression. Such a "syntactical macro" system can be nice because it's safer, and it's applicable in _most_ cases. But not for all - there are many situations where the full generality of C preprocessor macros (which are lexical / "token list" macros as you say) are useful.
X-macros might be the example that I use most, and that wouldn't work with syntactical macros.
Another application are macros around for loops, like FOREACH_FOO(...) or SCOPED_LOCK(...) for example. All dirty hacks that I like to use from time to time and that I prefer immensely to systems built in to the programming language and lead to immense complications in the language.
Another example would be partial lists of any kind, for example lists of compiler intrinsic attributes
#define FOO_API __attribute__((what1)) __attribute__((what2))..
printf("Hello from " PROG_NAME " version: " PROG_VERSION "\n");
#define DEFINE_BUILTIN(bkind, bname, num_args) else if (is_identifier(t->t_name.buf, bname)) \
...
#define PREC(t, p) case t: prec = p; break;
--
Expecting code to be so clean, always from the beginning, that all macros one would need ever can always be defined by syntactically complete expression bodys, that is not going to work. Just like there have been many attempts at getting rid of text-based programming languages and moving to structured (syntactical) editors - that hasn't panned out either.
This is the exact case where full macros can deliver a lot more power, more cleanly, than preprocessor macros. In this case, in Neat, you'd probably use a full parser macro rather than a call macro, and you could recognize arbitrary syntax, without allowing you to violate parenthesis order like C does.
Your macro can do whatever it wants, but it cannot conflict with other, preexisting syntax, such as (in C) defining half a loop or half a variable declaration. This way, it can fully own its syntax.
> Expecting code to be so clean, always from the beginning, that all macros one would need ever can always be defined by syntactically complete expression bodys, that is not going to work.
Correct, however the example syntax was a simplification of macros. Fundamentally, there's no reason a macro shouldn't be able to do anything that the compiler itself can do.
> Hopefully just bytes, which trivially allows storage of UTF-8.
Yes, it trivially allows, but then you'd need to deal with utf8-errors all over the program at runtime. Any method for string would need to be written such a way, that deals with invalid utf8 byte sequences.
Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. It gives a lot of edge cases which a programmer must bear in his/her mind constantly, because these edge cases would choose the least expected moment to jump on you. And then you'd need, for example, to invent a way to output an arbitrary byte sequence into a place where only UTF8 is allowed. I, personally, hate it. Like arbitrary byte sequences as a file names, even when I know that no one uses non-utf8 file names, I need to write programs working with file names in such a way, that allows arbitrary byte sequences, and to devise some syntax to output arbitrary byte-sequence into a terminal. Or into a web-page.
I see no good reasons to replace strings with arbitrary byte-sequences. If you need arbitrary byte-sequences then you have another abstraction for your task: an array. Much more powerful, because it can be array of bytes, of uint16_t, or of int64_t, or of your own struct. Why to spoil an abstraction of string with an ability to deal with arbitrary byte sequences?
> This should be solved in a string processing library
Then a type String also should be defined in a string processing library. Either String follows assumptions of a string processing library, or it doesn't. If it doesn't it makes the task of writing a good processing library much more difficult, and you'll get errors thrown from inside of a string processing library, and you'll need to deal with them.
UTF-8 was specifically designed such that most code can deal with it byte-by-byte without any decoding step. I've written many parsers for example, they all just read byte-by-byte and special things happen at ASCII characters (such as ';' or '\n'), and it works trivially with UTF-8 inputs, I don't have the care at all.
I've also written a text editor with a complicated text rope data structure in it. Do you think I should have made a different text rope for each different text encoding the editor should deal with?
No - what my editor does in UTF-8 mode for example, at the visual and editing layer it pulls out data byte-by-byte from the text rope and interprets it as UTF-8. If there is invalid UTF-8 it has to deal with it. But hey, that is the reality of files on a file system - they can contain encoding errors, right? Deal with them! (if you can't simply ignore them).
> Or when trying to get a slice with chars from 5 to 12.
There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called?
If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type.
The reality is that data is stored in memory in byte sequences, and that's the representation that a programming language should expose. Everything else is code / libraries.
No, I'd think that you should make a text rope for UTF-8, and then change encoding of text on the boundary. Or, it might be not an UTF8 but some other representation of Unicode, it depends. I see no reason to create a structure for an effective manipulation of strings without choosing a representation of a character at the compile time, because otherwise it would be slower than it might be. The more assumptions about your data you've made at the compile time, the less runtime conditioning you'd need, the faster your code would be.
Believe me, I had dealt with different encodings all the time. I'm Russian, and we had three widely used unibyte encodings for a cyrillic, plus different encodings for Unicode. So one had to deal with all of them all the time. The easiest way is to deal internally with Unicode only and to change encoding on the boundaries where your program communicates with an external world. There (on the boundaries) you can deal with errors, like character which cannot be represented in an output encoding (or cannot be represented in an internal one, but if you use Unicode it wouldn't be a problem). You can treat user input as an input in an external encoding and throw errors if she inputs something that cannot be encoded in an output encoding. It works all the time, while making your program able to deal with different internal encodings is a PITA, with errors thrown from the most unexpected places, with a spaghetti code trying to deliver errors to places where these errors could be sensibly dealt with.
> There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called?
You can use all of them, just pick distinctive names for them, like "char", "glyph", ... and any other you like. But when you did it, you'd want to know where are the boundaries of these things. You'd want to make slices of sequences of these things. If you cannot rely on a validness of underlying UTF8 then you'll be in trouble.
> If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type.
When I need to work with bytes, I use an array of bytes. Not a string, but an array of bytes. It was hard to grasp after years of experience with a C, but I did managed it at some point. Char is not a byte. Byte is not a char. Array is not a string, string is not an array. When I need an array to deal with bytes, I use an array. When I need a string to deal with characters, I use a string. It is a non-trivial idea for a C-programmer, because all his experience tells him that character and byte is the same thing. So if character is not a byte, then (he reasons) character doesn't exist.
If characters as codepoints is a too low abstraction for my task, I can create atop of it another abstraction dealing with glyphs, words, tokens, sentences or something. But the abstraction of codepoints must be a library feature, or I'd be forced to create it myself, to validate UTF8 all over the place, and so on. And if that so, then what the point to have an abstraction of string?
If characters as codepoints is a too high abstraction for my task, I can go lower and use an array of bytes.
It is really an easy idea, just C as a language tends to confuse people minds by teaching them that char==int8_t. At least my mind was confused and I managed to untangle that mess completely only around my 30th birthday. And several years later I've found that Rust's std is totally differentiate chars/bytes, strings/arrays as I do. I had fallen in love with Rust immediately.
But that's what I made. UTF-8 is encoded as bytes. I can store the bytes in the rope just fine.
The rope has a very simple API, basically read() and write() functions, just like a standard FILE I/O API. Do you want to pick on file system developers that they should add APIs for write_UTF8(), write_LATIN1(), write_KOI8(), write_BYTES(), etc.? And then go to network API designers to do the same for the socket I/O functions? And so on? Of course you don't do that, that would be very bad factoring.
And it's just the same for a rope API.
> The more assumptions about your data you've made at the compile time, the less runtime conditioning you'd need, the faster your code would be.
This is true in general, but the rope is just a storage. No processing happens there. The rope couldn't care less what things you store there. There is no point of having multiple identical read/write implementations.
But if you insist, I recommend to ask for an UTF-8 optimized HDD at your local computer shop :-)
> making your program able to deal with different internal encodings is a PITA
If your program has to deal with multiple external encodings, either you can convert at the boundaries to a canonical internal encoding, or you can't in which case it probably becomes a little more work since you have to convert at different places.
This has nothing to do with what I said, though.
In this paragraph you seem to be confusing the C's "char" with the much more fuzzy idea of "Character" which has like 13 valid definitions.
C's "char" is abstractly defined as the smallest addressable unit of memory available on the machine (required to have at least 8 bits), and historically there have existed 8-bit, 9-bit, 16-bit, or even 36-bit chars. In today's practice it is universally taken synonymous for (8-bit) bytes since all hardware is 8-bit by now. Some people like to be pedantic about the distinction between byte and char, but I most often do not, especially since char is the generally interoperable type in C (with respect to type punning etc.), while uint8_t to my knowledge is not.
"Character" is sometimes understood as "Unicode codepoint" (typically represented as a 32-bit entity, or even as a UTF-8 encoded slice of bytes) or in some cases understood as "Unicode glyph" (probably represented as a slice of codepoints), sometimes understood as even other things.
This stuff is specifically some of the big design wins of these systems. I recommend you to peek over at the Win32 API for example, with its myriads of *A and *W functions, required compiler settings and macro magic to switch between those (even though of course they can't really paper over the difference), then best practice recommendations that have changed multiple times in history, then strange bugs that appear when some invalid data has sneaked into a system that was assumed "pristine"...
Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.
And a difference between "UTF-8" and arbitrary bytes is insofar sensible in that you can perform text operations on the former, but not on the latter. Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
See here for example: https://docs.microsoft.com/en-us/windows/apps/design/globali...
If you browse around various documentation, you'll see contradicting statements which variants are recommended, which I take as another sign that the idea of making a distinction on the type level is simply a bad idea.
As to the macros, yes, if you give the A or W explicitly, you won't need the macro setting that translates the unspecific names to the A or W variants. And as said, the macros aren't a good idea anyway, as it's still extremely hard to properly abstract the distinction (the types are different sizes!!), so code is generally tied to a specific choice either way.
> Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
It can be stored as a byte stream. Whenever you work with the data, you might need to operate on transformed representations - 32-bit codepoints, or larger combinations, or even more complicated stuff like words, sentences, paragraphs, tags, whatever it needs to do the task as done. This is programming, you transform data to achieve things.
What I say is that it's stupid to make a distinction on the type level between things that are entirely the same thing in memory, and that are going to be used for the same things. It's stupid because it unnecessarily create incompatibilities between data and introduces unnecessary "conversions" / copies and requires more code.
Secondly: if there are worse places than Unix, it doesn't mean that Unix is good.
I mean, I love Unix, but Unix sometimes a nightmare to deal with. And kernel believing that "string" means "a byte array with a zero-terminator" one of the worst things of Unix. If one needs an array of variable size, it might do either:
struct unsized_array_t {
size_t size;
uint8_t bytes[]; // being really a bytes[size]
}
struct array_pointer_t {
size_t size;
uint8_t *ptr;
}
The first one is tricker, you'd need to pass around pointers to it, and slices of it would be a problem. But at least you wouldn't need to scan memory to learn the size of it.
Unix strings are not strings but zero-terminated arrays of non-null bytes. It would be obvious if you try another way to deal with a variable size. The Unix-Fathers had an idea how to deal with string, but their idea was proven bad. A lot of functions they invented to deal with their "strings" are now deprecated or even forbidden. Like `gets` for example. I'd hate to use strcpy, I'd better use strncpy, and other functions with `n` inside. But if you started to juggle not just with pointers to strings but with those n's also, you would think of representing strings as structs with an embedded n. And -- viola -- all the Unix's string crap goes through a window. You end with variable sized arrays, and you doesn't need strings anymore. Without losing anything useful.
> Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.
I like this one. Maybe you are right after all.
C doesn't prevent you from creating such a string type, or just passing a separate length as argument to functions. In fact it's the right thing to do (pointer + length) in many cases.
All C does is offer you string literals that are zero-terminated, which is typically practical. Often those are all you need, i.e. to store a plain-text identifier that doesn't allow for NUL characters anyway - so NUL can be used as a sentinel for efficient storage.
Then there is the C standard library, which is a bag of bad practice. Just ignore 90% of the stuff in there. stuff like strtok() etc. is bad. Mostly you want to use memcpy(), memcmp(), strcmp(), maybe strcpy()/strncpy() etc. Then there's stuff like malloc()/free() and stdio and especially the formatting functions that you can use to get something up and running before possibly replacing them with something better. Well, that's about it:-)
See here, it might be that they recognized that choices made in the 90's were wrong: https://docs.microsoft.com/en-us/windows/apps/design/globali...
"Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes."
> I like this one. Maybe you are right after all.
Thanks :-)
I don't know, maybe for interoperable type safe types to use across an ecosystem of libraries without wasting CPU cycles converting among them?
This is especially true for the "to use across libraries without wasting time on conversions". I've said it many times, plain C interfaces (pointer + length, or slices if you insist but I don't like them because they are a less normalized representation) ... are the best way to design interfaces optimizing for interoperability. No need for any pointless conversion, just tell the API where your data is located. The physical fact that is needed for communication is the memory (address + length), it's the necessary and sufficient information to carry out the task.
Yes, nowadays "safety" is not just about Out-of-bounds accesses but people expect the system to even protect against resource leaks, double-free, user-after-free, and race conditions. But even when it is the goal to machine check this by introducing a system that requires thinking on the small scale in isolated mini-units ("classes"/"types") - is there a point in locking in on a specific implementation of dynamic arrays? (Not a rhetoric question)
Maybe we should tell them to stop if they are so good.
Also, they have loads and loads of money and their jobs come with prestige, so they have no problem attracting developers to jobs that are perceived by some programmers (such as me) as boring boilerplate jobs that make me miserable.
That answer was more related to the dynamic arrays discussion. If you want to move to hardware memory tagging, is that even a big thing? In any case my understanding is that it would work with pointer + length just as well, because the hardware tags are created at buffer allocation time, not based on arguments passed to a function.
So lots of money is being burned to ensure that C code is caged and does no harm, in scenarios where C is to still be used.
Hopefully just bytes, which trivially allows storage of UTF-8.
> built-in dynamic arrays, yess.
Dynamic arrays are only few lines to implement. There are different ways to do them, and no matter how, there will always be some problematic aspects. Not sure why you would want to choose one particular implementation and elevate it to a higher status.
> No preprocessor. Heck yes, it's a crutch.
It's ugly and inexperienced users will write bugs using it, but it's also tremendously useful. You mentioned quasiquoting as an alternative, but I'm not positive that it works as a preprocessor replacement for a language that lacks the "homomiconity" of LISP. Are there examples that show it works?