News for hackers. Stuff that matters.
Re:Curl is C by tannhaeuser on Monday March 27, 2017 @08:17AM (#13967038)
Not only is curl based on C, but so are operating systems, IP stacks and network software, drivers, databases, Unix userland tools, web servers, mail servers, parts of web browsers and other network clients, language runtimes and libs of higher-level languages, compilers and almost all other infrastructure software we use daily.

I know there's a sentiment here on HN against C (as evidenced by bitter comments whenever a new project dares to choose C) but I wish there'd be a more constructive approach, acknowledging the issue isn't so much new software but the large collection of existing (mostly F/OSS) software not going to be rewritten in eg. Rust or some (lets face it) esoteric/niche FP language. Even for new projects, the choice of programming language isn't clear at all if you value integration and maintainability aspects.

Re:Curl is C by wyldfire on Monday March 27, 2017 @09:35AM (#13967553)
> I know there's a sentiment here on HN against C

I think there's two major against-C groups: those of us who have worked with C for decades and those who never worked with it. I'll try and speak for those of us who've used it for decades. The popular high-level languages that have arrived since ~1995 (Java, Python, JS, C# and friends) are excellent productivity increases. In general, they sacrifice memory and performance in favor of robustness and security. For enormous software problem domains, we just don't need C's complexity or error-proneness.

Until Rust, there's been very close to zero serious competitors for C if I wanted to write a bootloader, OS, or ISR. Not even C++ could do those (without being extremely creative on how it's built/used). The ~post-2000 languages (golang, swift, D etc) can't do that (perhaps D's an exception but it wasn't an initial goal AFAICT). This is huge, IMO.

We've groaned and grumbled about how hard it is to parse C/C++ code for decades. This is a big deal for tooling. Because of the language's design, even if you use something "simple" like libclang to parse your code, you still have to reproduce the entire build context just to sanely make an AST. All of those other new languages above probably address this problem but also add all kinds of other stuff which we can't have for specialized problem domains (realtime/low-latency requirements, OSs, etc).

> collection of ... software not going to be rewritten in eg. Rust or some (lets face it) esoteric/niche FP language

IMO it's not appropriate to lump Rust in with "nice FP language"s. And don't look now but lots of stuff is being rewritten in Rust. Fundamental this-is-the-OS-at-its-root stuff: coreutils [1], "libc" [2], kernels [3], browser engines [4].

[1] https://github.com/uutils/coreutils

[2] https://github.com/japaric/steed

[3] https://github.com/redox-os

[4] https://github.com/servo/servo

Re:Curl is C by tannhaeuser on Monday March 27, 2017 @09:58AM (#13967734)
> IMO it's not appropriate to lump Rust in with "nice FP language"s.

Maybe I should have expressed it better, but I didn't intend to lump these together.

>And don't look now but lots of stuff is being rewritten in Rust.

I'm myself cautiously optimistic re Rust, but having been burnt by C++ in the past I'm not enthusiastic about fighting language idiosyncrasies (though modern C++ certainly deserves a second look). Then there's the issue (some might argue it's a plus) that Rust is at the same time a language, a lib, and the only compiler implementation (unlike C or C++ which give you choice).

Re:Curl is C by cat199 on Monday March 27, 2017 @05:54PM (#13972154)
> Then there's the issue (some might argue it's a plus) that Rust is at the same time a language, a lib, and the only compiler implementation

This was essentially the case for C and C++ as well for somewhere around 5-15 years a piece, depending on how you measure it.

Re:Curl is C by nwmcsween on Monday March 27, 2017 @03:34PM (#13970927)
The rust coreutils is an excellent example of the issues of having such a mess of abstractions, the resulting binaries are literally magnitude larger than busybox equivalents.
Re:Curl is C by wyldfire on Monday March 27, 2017 @04:52PM (#13971665)
> issues of having such a mess of abstractions, the resulting binaries are literally magnitude larger

They're significantly larger, yes -- it's a fair complaint of rust. But it's mostly because of static linkage AFAIK [1] and not "a mess of abstractions".

[1] https://github.com/uutils/coreutils/issues/747

Re:Curl is C by nwmcsween on Monday March 27, 2017 @06:25PM (#13972376)
And what do you think pulls in all the symbols for static linking? Maybe the mess of abstractions perhaps?
Re:Curl is C by ssokolow on Monday March 27, 2017 @09:54PM (#13973404)
Actually, the culprit is Rust's decision to statically link its standard library and all its dependencies by default.

Things like libunwind, libbacktrace, embedded debugging symbols for backtraces, and the jemalloc allocator aren't free.

If you ask for dynamic linkage (with the caveat that Rust doesn't have a stable ABI yet), you get a ~8K Hello World binary.

It's also possible to prune down the statically-linked size by opting out of various conveniences like jemalloc. (They're working toward making the system allocator default but don't want to regress Servo in the interim.)

...and if opt into static linking with GCC and G++ (and ask Rust to make its link to libc static), Rust can actually outdo them on a Hello World.

Here's a detailed exploration: https://lifthrasiir.github.io/rustlog/why-is-a-rust-executab...

Re:Curl is C by nwmcsween on Monday March 27, 2017 @10:20PM (#13973538)
> Actually, the culprit is Rust's decision to statically link its standard library and all its dependencies by default.

No it really isn't, static linking does not imply bloat as commonly perpetuated.

> It's also possible to prune down the statically-linked size by opting out of various conveniences like jemalloc

Try this: opt out of everything except the standard library, create something somewhat trivial and idiomatic in both rust and c, compile and see what you get.

> Rust can actually outdo them on a Hello World.

Hello word is hardly a use of the standard library.

Re:Curl is C by ssokolow on Tuesday March 28, 2017 @07:23AM (#13975661)
> No it really isn't, static linking does not imply bloat as commonly perpetuated.

I never said it implied bloat. I said that, if you ask Rust to link dynamically despite the lack of a stable ABI, you'll get binaries of a size similar to C and C++.

> Try this: opt out of everything except the standard library, create something somewhat trivial and idiomatic in both rust and c, compile and see what you get.

I'll need you to be a bit more specific than "somewhat trivial", given that "Hello world" uses println! or printf() but you consider it ineligible.

> Hello word is hardly a use of the standard library.

println! aside, it's a data point and that's all I meant by it.

Re:Curl is C by Manishearth on Monday March 27, 2017 @11:47AM (#13968679)
Could you expand on why you think Rust is a serious C competitor for OS/ISR/bootloaders but not C++? This statement intrigued me.

I thought C++ had naked functions and all the things you need to write an OS.

Re:Curl is C by Rusky on Monday March 27, 2017 @12:35PM (#13969190)
Not wyldfire, and I think that claim is a mischaracterization, but the main obstacle to using C++ in the kernel is that some of its language features require runtime support (new/delete, globals/statics with constructors, exceptions).

You can of course just ignore those when writing kernel code- they get ignored in application code much of the time! But I suppose at that point it could be argued that you're just writing C with a C++ compiler?

Re:Curl is C by Manishearth on Monday March 27, 2017 @01:00PM (#13969475)
I mean, if you're writing a kernel in Rust you have the same issue. In that case you'd use no_std, which takes away the part of the stdlib that depends on allocation and such (also threads and other niceties).

You can lose new/delete and .bss statics and still write reasonable, even "safe", C++. Rust doesn't have .bss statics by design (lazy_static emulates this for you though). new isn't necessary for the "modern" C++ safety stuff and you can write pretty good modern C++ without new. All new gets you is a nice wrapper around allocation, and when writing a kernel you can't and shouldn't allocate anyway. In Rust, too, you would not be allocating, either via memmap/malloc or via Box::new().

So it wouldn't be "C with a C++ compiler", it would be "C++ without allocations", which is a restriction from the problem statement anyway.

Re:Curl is C by 0xFFC on Monday March 27, 2017 @10:19PM (#13973536)
I don't get it. AFAIK you can implement all of those things in your abstraction, and then use it like canonical C++. I think you are wrong. Please correct me.
Re:Curl is C by kyberias on Monday March 27, 2017 @03:33PM (#13970913)
Re:Curl is C by pjmlp on Tuesday March 28, 2017 @03:20AM (#13974716)
Add BeOS, OS X drivers (IO Kit) and OS / 400 after they started rewriting the PL/M code, to that list.
Re:Curl is C by wyldfire on Monday March 27, 2017 @12:31PM (#13969137)
I might have to walk that back. It seemed to me that no_std was "more straightforward" and/or "more formalized" than "#pragma interrupt" (etc). But I could be wrong there -- if so, mea culpa (the post is no longer editable).
Re:Curl is C by Rusky on Monday March 27, 2017 @12:39PM (#13969239)
Rustc did recently get a "x86-interrupt" calling convention, but that's unrelated to #[no_std], and only works on x86. Either way, "#pragma interrupt" should work just as well in C++ as in C, since C++ doesn't really change any aspect of the language that matters there.

Further, even in C I rarely see use of "#pragma interrupt"-like tools- rather, everyone still seems just to use per-platform assembly glue code. (To be fair, my experience is mostly in kernel code for things like Linux, rather than standalone embedded applications where "#pragma interrupt" would be more valuable.)

Re:Curl is C by Manishearth on Monday March 27, 2017 @01:09PM (#13969572)
no_std is more formalized, though C++ enforces the same thing by failing to link if you try using malloc (or whatever) when writing a kernel. no_std also means that it's very easy to tell if a crate works without the stdlib, so you can use code from the ecosystem instead of rolling your own.

Ultimately the Rust OSes resort to some handwritten assembly as well. I think that's going to be a constant of writing a kernel. Rust is working to minimize it (e.g. with things like `extern "x86-interrupt" fn`), but at a kernel level there are just some kernel specific asm instructions (like all of the TLB stuff) that either compiler will probably never support generating without inline asm.

So while Rust may be better than C++ at writing OSes (I'm not sure! I haven't looked at all the stuff you need to write an OS in C++), I do think they're in the same ballpark, close enough that if Rust is a "serious" competitor C++ probably is too :)

Re:Curl is C by zeveb on Monday March 27, 2017 @10:48AM (#13968145)
> Until Rust, there's been very close to zero serious competitors for C if I wanted to write a bootloader, OS, or ISR … We've groaned and grumbled about how hard it is to parse C/C++ code for decades.

I honestly think that Common Lisp can do this quite well. It was designed to be a high-level language, but it's completely capable of working at the machine level, pleasantly and easily. Unlike C, most of the time one has safety, but one can disable safety when necessary with a simple (declare (safety 0))).

Performance is extremely good with modern compilers, although I don't know how good they would have been back in the old days.

And it makes parsing the language dead simple. Here's the entire reader algorithm: http://clhs.lisp.se/Body/02_b.htm

From what I can tell of Rust, it doesn't look easier to parse than C (but I've not looked deeply); certainly, it's orders of magnitude more difficult to parse than Lisp.

I believe that Standard ML or OCaml could do similar things as well, albeit at the cost of being more difficult to parse. Smalltalk is maybe a little less capable, but somewhat easier to parse.

They've all been around for decades.

Re:Curl is C by Manishearth on Monday March 27, 2017 @11:52AM (#13968729)
Yes, it's harder than lisp, but it's still much easier than C. C and C++ have issues due to ambiguities that make them context-sensitive. C++ has it worse because parsing is dependent on typechecking because of templates.

Rust is not 100% context-free, but the feature that is non-context-free (raw strings, a rarely used feature) is still pretty easy to parse, and even if you capped it at 6-level raw strings you'd probably be able to parse all the Rust code out there.

Re:Curl is C by wyldfire on Monday March 27, 2017 @11:15AM (#13968373)
> I honestly think that Common Lisp can do this quite well

I haven't used any lisp dialects for decades, so I have naive questions: is there really sufficient support from compilers+linkers to write a bootloader in lisp? Do I have to do a lot of bootstrapping in assembly to bring up lisp interpreter before I can execute the lisp code or does the ahead-of-time-build result in executable machine code? Can I do inline assembly (not required but a really key benefit IMO)? Are there numerous examples where someone's already written one in lisp?

Re:Curl is C by tjalfi on Monday March 27, 2017 @02:33PM (#13970411)
https://github.com/dym/movitz is a Common Lisp system that runs on bare metal x86. The source code is quite readable.

The rest of this post is an excerpt from an email I sent 6 years ago.

The following comments on runtime systems are partially based on a long c.l.l thread with posts by Lucid, Symbolics, and Franz alumni.

Franz uses a 3-layer approach: CL, a low-level Lisp, and C.

Lucid started with Lisp that generated assembler but reluctantly added some C.

Symbolics Lisp Machines used bootstrap code in a Pascal-level language with prefix syntax. A Symbolics alum said that in retrospect they should have used C.

Most Lisp implementations have subprimitives - low-level functions that can circumvent the type system, often with a prefix such as % or :.

Assembly language integration dates to Lisp 1.5 and there are several common approaches.

1. turn the optimizer off - this is easy to use and implement.

2. optimize the assembler block - Naughty Dog GOAL did this.

3. annotate the assembler block with pragmas that indicate side effects. This can be error prone and difficult to use. https://www.pvk.ca/Blog/2014/08/16/how-to-define-new-intrins... is an example of this approach.

Edited to add the SBCL intrinsic link.

Re:Curl is C by kazinator on Monday March 27, 2017 @03:24PM (#13970836)
3 sounds like where GCC got its inline asm concept from: annotate the assembly with what are the inputs and output operands with constraints (do they have to be certain kinds of registers), and whether anything has surprising side effects.
Re:Curl is C by sanxiyn on Monday March 27, 2017 @12:11PM (#13968910)
For an example of operating system kernel completely written in Lisp, see https://github.com/froggey/Mezzano
Re:Curl is C by smitherfield on Monday March 27, 2017 @10:08AM (#13967793)
Parsing is IMO of all the complaints you could make about C/++ rather bikesheddy. Parsing is a solved problem. Modern compilers can parse millions of LoC per second. And most of the specific parsing-related complaints (pointer dereference or multiplication?) about C/++ are also true of Rust. (Edit: Nope, brain fart on my part, see below). And, AFAIK, all C/++ parsing is well-defined, if counterintuitive in certain edge cases.
Re:Curl is C by steveklabnik on Monday March 27, 2017 @10:37AM (#13968051)
> most of the specific parsing-related complaints (pointer dereference or multiplication?) about C/++ are also true of Rust.

This should not be true, and we fought hard to keep it that way. There's one spot of Rust's grammar that's context-sensitive, for something very rarely used, and other than that, it's all much simpler.

Re:Curl is C by smitherfield on Monday March 27, 2017 @10:57AM (#13968218)
You're right. AFAIK types and identifiers are always unambiguous in Rust. I was thinking visually (same operator) instead of in terms of specification and implementation. Shows me to make flippant comments from the toilet!

My larger point is that there are plenty of very good reasons to criticize C/++, and parsing is a minor one since parsing is fast, and even if the creation of the AST isn't context-sensitive, verifying its correctness (is this identifier in scope?) still is.

What is the context-sensitive spot in Rust's grammar?

Re:Curl is C by wyldfire on Monday March 27, 2017 @11:11AM (#13968331)
> plenty of very good reasons to criticize C/++, and parsing is a minor one

Ok, fair bit, it's a frustration for me but admittedly not as important as the other differences.

I mention it because it's a wart in C's language design and I figured Rust's safety features are already well-known and heavily discussed. If I want to write a simple tool "ask this tree of .c files how often they use an identifier with name 'X' or type 'Y'", I have to find out the include paths, defines, all kinds of other "noise" just to find out what could be a relatively simple query of the source base.

Re:Curl is C by Manishearth on Monday March 27, 2017 @12:21PM (#13969016)
This also means that autocomplete tools usually need to be taught how to build a project. YCM has this whole conf file where you specify the header locations and stuff and it's like rewriting half the makefile.
Re:Curl is C by steveklabnik on Monday March 27, 2017 @10:59AM (#13968234)
Re:Curl is C by madphrodite on Monday March 27, 2017 @10:54AM (#13968204)
No competent C programmer is going to switch to rust based on the projects you've just mentioned. The comments in the commits speak for themselves.
Re:Curl is C by wyldfire on Monday March 27, 2017 @11:18AM (#13968397)
Please, connect the dots for me. An initial skim of the commit messages did not yield any egregious "Utter Disregard for Git Commit History" [1]. Even if it did, it may just mean that the maintainer is focused more on results and robustness than preserving a pristine history of the project.

[1] https://zachholman.com/posts/git-commit-history/

Re:Curl is C by madphrodite on Wednesday March 29, 2017 @11:52AM (#13987455)
Don't think that was where I was going. It was more of the none of this stuff is really done, it is hard in this language and help would be appreciated..as understood through the project splash page and then examined through commits?
Re:Curl is C by mundanevoice on Monday March 27, 2017 @10:16AM (#13967857)
I hope Rust doesn't face the same fate as other ambitious projects by Mozilla. Rust has a quite unusual syntax compare to any other systems programming language. Also, there is a big learning curve. Keeping all the benefits aside, I really hoped Rust had a simpler syntax. I really think, one day a language will borrow the good parts of Rust with a simpler syntax and get ahead of it. Rust in its current form will never be as successful as C/C++.
Re:Curl is C by naasking on Monday March 27, 2017 @12:03PM (#13968845)
> Rust has a quite unusual syntax compare to any other systems programming language.

In what way? It's very C-like in its basic syntactic structure, just more regular.

Re:Curl is C by mundanevoice on Tuesday March 28, 2017 @12:49AM (#13974164)
Well, I was not talking about basic syntactic structure. For eg: take a look at this https://github.com/servo/servo/blob/master/components/conste...

I am not against Rust. Rust has some great ideas and intent. I just feel they should have created simpler syntax. A more complex and unusual syntax doesn't have any real benefits IMO.

Re:Curl is C by naasking on Tuesday March 28, 2017 @12:00PM (#13977725)
You still haven't pointed out what syntax is problematic exactly. I've never programmed in Rust, but I don't have any trouble reading it coming from a C and C# background.

What exactly is the problem with the example you linked to?

Re:Curl is C by pcwalton on Tuesday March 28, 2017 @12:04PM (#13977760)
What's the issue with that example?
Re:Curl is C by naasking on Tuesday March 28, 2017 @12:50PM (#13978247)
Not having programmed in Rust, the fact that Rust requires all type parameters to be used, thus ruling out proper phantom types, was semantically surprising to me. But I don't understand what syntactic issue the other poster had with that example.
Re:Curl is C by steveklabnik on Tuesday March 28, 2017 @01:11PM (#13978429)
Re:Curl is C by naasking on Tuesday March 28, 2017 @01:46PM (#13978792)
Yes I get it, it just seems unnecessary to require every type parameter to be used. Do you have a link to the discussion motivating this choice?
Re:Curl is C by steveklabnik on Tuesday March 28, 2017 @02:17PM (#13979148)
In my understanding, it has to do with variance. This happened a very long time ago, before 1.0, and so I don't know where the discussion happened, off the top of my head.
Re:Curl is C by naasking on Tuesday March 28, 2017 @03:17PM (#13979724)
Found it: https://github.com/rust-lang/rfcs/blob/master/text/0738-vari...

It still seems bizarre to me that a purely type-level expression is forced by an effectively non-existent term. That RFC specifically states that the main problem is that the results of variance inference are largely erased by assuming invariance. That seems like a sensible default for unused type parameters too.

It seems from the conclusion of that post that PhantomData only survived because this was the smallest change they had to make to get this all to work better, and because some of this PhantomData could be used for other analyses in the compiler (although it's not clear if better type information could have replaced these uses anyway).

Re:Curl is C by pcwalton on Monday March 27, 2017 @12:06PM (#13968869)
Can you elaborate as to the specific syntax problems you see with Rust?
Re:Curl is C by mundanevoice on Tuesday March 28, 2017 @12:52AM (#13974179)
This is from servo codebase - https://github.com/servo/servo/blob/master/components/domobj...

Take a look at this. It isn't as simple and clear as C/C++ for example and requires significant investment in parsing and understanding the code.

Re:Curl is C by mkawia on Tuesday March 28, 2017 @02:45PM (#13979431)
That's quote ,a library that generates code for you at compile time ,it takes code as input,has it's own syntax. It's like compile-time reflection.

It's not code rust programmers would normally write, I'm one of those programmers. I'm glad some libraries like serde,rocket and diesel are using it to generate code instead of doing run-time analysis.

Re:Curl is C by pcwalton on Tuesday March 28, 2017 @12:02PM (#13977740)
You quoted a macro definition. The equivalent in C++ would be heavy preprocessor use, which is usually much worse.
Re:Curl is C by sullyj3 on Monday March 27, 2017 @11:37PM (#13973840)
The type and lifetime annotations can get pretty confusing.
Re:Curl is C by pcwalton on Tuesday March 28, 2017 @12:03PM (#13977746)
Do you have a specific proposal as to what lifetime syntax should be changed to?
Re:Curl is C by fiedzia on Monday March 27, 2017 @09:52AM (#13967688)
> the large collection of existing (mostly F/OSS) software not going to be rewritten in eg. Rust or some

It is happening and will keep happening and it is really necessary at some point. Sure, I don't expect large project to be rewritten overnight, but every large project is being redone eventually. Especially when C becomes main source of problems. And you can introduce better languages gradually.

> operating systems, drivers

https://github.com/redox-os/redox

> IP stacks

https://github.com/QuiltOS/QuiltNet

Many of those are often written in userspace, where you are free to use any language. Several of them will provide sufficient performance.

> databases, Unix userland tools, web servers, mail servers, parts of web browsers and other network clients, language runtimes and libs of higher-level languages, compilers and almost all other infrastructure software we use daily.

For all of those you'll find Rust implementations. Some are work in progress, some are already widely used.

Re:Curl is C by pjmlp on Monday March 27, 2017 @08:31AM (#13967118)
Historical accident, I started to work back in the days when UNIX was the only OS implemented in C.

Everything else outside UNIX was using Assembly, Algol, PL/I, Modula or Pascal dialect.

C owes its success to UNIX's adoption by the market, as operating system available almost free of charge to universities, with source code available.

Re:Curl is C by davexunit on Monday March 27, 2017 @10:03AM (#13967763)
C was originally a watered down version of BCPL because the computers they were targeting had very limited capabilities compared to other computers of the time. The first version of C didn't even have floating point numbers.
Re:Curl is C by pjmlp on Monday March 27, 2017 @11:48AM (#13968692)
Meanwhile in 1961 we had ESPOL, followed by NEWP, nowadays living on the Unisys MCP ClearPath mainframes.
Re:Curl is C by erelde on Monday March 27, 2017 @08:23AM (#13967068)
I am student and I like C, I've tested Rust/Go, I like the feeling of C. Maybe that sentiment will change later, but for now, I like C. It's simple and sharp and there's lots of doc/books.
Re:Curl is C by IshKebab on Monday March 27, 2017 @09:38AM (#13967581)
C seems relatively simple, but it's also incredibly easy to do the wrong thing usually without realising it. That's the problem. I suggest you read up about 'undefined behaviour'.
Re:Curl is C by stevedonovan on Monday March 27, 2017 @10:22AM (#13967909)
'sharp' would be a good word ;) There's lots to like about it but writing good safe C is harder than it looks.
Re:Curl is C by floatboth on Monday March 27, 2017 @09:26AM (#13967490)
Have you tried doing string manipulation in C? ;)
Re:Curl is C by asveikau on Monday March 27, 2017 @10:41AM (#13968088)
Not OP, but I actually think string manipulation in C is really elegant. Many people who complain about it have too many allocations in their code and are trying to port the allocation-heavy non-C way of thinking to C. The C way I know focuses mainly on character at a time iteration with emphasis on not copying the source string.

I'm reminded of a time a colleague needed something like string.split, and working in c++ he filled a std::vector<std::string> with the result. Using a more C way he'd really only have needed a couple of pointers on the stack.

Re:Curl is C by imron on Monday March 27, 2017 @11:03AM (#13968268)
> The C way I know focuses mainly on character at a time iteration

Ah, yes, the good 'ole C way that only works reliably in English.

Re:Curl is C by dbaupp on Monday March 27, 2017 @11:12AM (#13968343)
It's a little naïve to think it works on English without the coöperation of the users. (Also, less glibly, things like emoji seem to be becoming more and more popular.)
Re:Curl is C by imron on Monday March 27, 2017 @11:34AM (#13968566)
Indeed. In fact as we speak I am fixing a bug related to string handling with emojis.
Re:Curl is C by asveikau on Monday March 27, 2017 @11:57AM (#13968778)
Funny.

You can parse utf-8 character at a time. Some characters advance the pointer by 4 at an iteration and some less.

Re:Curl is C by imron on Monday March 27, 2017 @08:33PM (#13972992)
You can, and then you get a 4-byte long character 1-byte before the end of your data, you skip over the null-terminator and into the stack, and bang.

Yes, you can avoid this if you're careful and you understand the intricacies of utf-8 (or some other multi-byte encoding), but it very quickly stops being elegant.

Re:Curl is C by dbaupp on Monday March 27, 2017 @12:19PM (#13968991)
What do you mean by "character"? If you mean code point or "unicode scalar value", sure, but if you mean user-visible character (grapheme), it's much more complicated: even something "simple" like ö could be one or two code points.
Re:Curl is C by asveikau on Monday March 27, 2017 @12:40PM (#13969243)
I mean your iterator is char* and you advance it by adding. That's it.

I do NOT mean that char itself corresponds to a glyph or codepoint, you are seriously preaching to the choir making that lecture to me.

Re:Curl is C by fauigerzigerk on Monday March 27, 2017 @03:41PM (#13971002)
>you advance it by adding

And when do you stop? UTF-8 strings can have zero bytes in them so treating them as C strings is potentially error prone depending on the context.

Re:Curl is C by imron on Monday March 27, 2017 @08:52PM (#13973084)
> UTF-8 strings can have zero bytes in them

This is not true. A zero-byte in a utf-8 string is the null-terminator and utf-8 strings can be treated exactly like C strings in terms of where the string ends.

What you do need to look out for is malformed utf-8, for example, 1 byte before the null terminator you get a lead byte saying the next character is 4-bytes long.

If you're not checking each byte for null and just skipping based on the length indicated by the lead byte then you're in for a crash.

Where utf-8 strings differ from C strings is slicing. You can't just slice the string at some random point without doing extra validation to make sure you only slice on codepoint boundaries.

Re:Curl is C by dbaupp on Monday March 27, 2017 @09:13PM (#13973174)
> A zero-byte in a utf-8 string is the null-terminator and utf-8 strings can be treated exactly like C strings in terms of where the string ends.

No, the parent was correct: UTF-8 encodes NUL (i.e. \0) as a single zero byte (e.g. in contrast, Modified UTF-8[1] uses an overlong for NUL, so there's never any possibility of an internal zero). Of course, an application/library can choose to restrict itself to only handling UTF-8 that doesn't contain internal NULs, but the spec itself allows for zero bytes in a string.

[1]: https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8

Re:Curl is C by imron on Tuesday March 28, 2017 @12:33AM (#13974098)
We are in agreement that the only time a single zero byte can be found in well-formed utf-8 is for the NUL character.

By definition, with a null-terminated string, NUL is the terminator.

If you want to have strings that contain NUL, then by definition you can't use a null-terminated string.

This is true of utf-8 or regular C strings.

Re:Curl is C by fauigerzigerk on Tuesday March 28, 2017 @04:06AM (#13974919)
The point is, if you handle strings the C way, you're not in conformance with UTF-8.

If someone passes you a text file that is verified to be valid UTF-8 and contains, say, access permissions, then you better not stop parsing it at the first '\0' character.

None of this is a huge problem, but it's something to be aware of. C string handling is incompatible with UTF-8.

Re:Curl is C by imron on Tuesday March 28, 2017 @04:38AM (#13975033)
File processing and string processing are not the same. If you have a file that has a specific data format outside of the encoding, and that format includes NUL bytes as part of the data, then obviously process the file based on that format.

That's separate from string handling.

UTF-8 was originally designed to be compatible with NUL terminated strings and keep NULs out of well formed text.

In fact it was the first point in the 'Criteria for the Transformation Format', mentioned in the initial proposal for utf8.

https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

Re:Curl is C by fauigerzigerk on Tuesday March 28, 2017 @05:09AM (#13975155)
>File processing and string processing are not the same

The UTF-8 spec doesn't make that distinction as far as I know. There's a simple fact: A valid UTF-8 byte sequence can contain nul characters. So you can't naively use C string handling functions on it. And as someone else has correctly pointed out, the same is true for ASCII.

I'm just pointing out a potential pitfall and a source of security issues. Some might assume that after validating UTF-8 text input, you could just dump it in a C string and process it using C's string functions. But that's not the case.

Re:Curl is C by asveikau on Monday March 27, 2017 @06:09PM (#13972265)
Unless you have U+0000 there isn't any other sequence of code points that has an 0x00 byte in UTF-8. I don't see this as a huge problem.

If you really do need it there are some C language libraries that use "pascal-ish" structs to do strings. UNICODE_STRING in Windows comes to mind. Doing strings in C doesn't force you to use C strings, it's just the most common thing to do.

Re:Curl is C by fauigerzigerk on Tuesday March 28, 2017 @04:08AM (#13974923)
No it's not a huge problem, but if you're not aware of it, it could easily lead to a security breach: https://news.ycombinator.com/item?id=13974919
Re:Curl is C by MichaelGG on Monday March 27, 2017 @06:27PM (#13972390)
It's the same for ASCII - UTF-8 zero byte is NUL.
Re:Curl is C by pikzen on Tuesday March 28, 2017 @11:52PM (#13983249)
What are combinators?

Go parse some zalgo with your 4 per iteration algorithm. I'll be there, waiting and laughing.

C string handling is not elegant, nor does it fit the realities of the world.

Re:Curl is C by cat199 on Monday March 27, 2017 @06:05PM (#13972244)
So it's somehow C's fault that Unicode uses variable-length encoding, which is automatically going to be harder to process correctly at a byte-by-byte level than a fixed-length method, and also included known-C-incompatible null bytes?
Re:Curl is C by imron on Monday March 27, 2017 @08:27PM (#13972972)
> So it's somehow C's fault that Unicode uses variable-length encoding

Parent said string handling in C was elegant. My point is that it becomes fraught with (even more) issues once you throw non-English language at it.

It is C's decision to handle strings in this way, and the decision of many C programmers to treat all strings as if they are just iterable character pointers.

It's a recipe for bugs.

Re:Curl is C by asveikau on Monday March 27, 2017 @10:46PM (#13973636)
I am the parent you are talking about. I've made this argument many times with people: Unicode is crazy complicated in any programming language. People think that widening the char width will help - well you seem to be somebody who knows Unicode so you probably know the horrors of surrogates, combining characters vs. pre-composed diacritics, zero-width joiners, Han unification, variation selectors, BiDi... This is in no way just a C thing to deal with all that nonsense. I've not seen any language or library that I'd say does it "well" and saves individual programmers from considering the above. They all punt the issue to the programmer.

I've heard (mostly here) that Swift does something different and treats glyphs as the basic unit. I haven't had a chance to look at precisely what that does. Given all the issues I've seen elsewhere I'm skeptical that someone, anyone can pull that off correctly.

UTF-8 at least has one elegance (there's that word again) in the design in that you can do some "dumb" ASCII things and if your code does not know what to do with fancy unicode, you can check the high bit of any given octet and "safely" skip over it and any adjacent nonascii sequence if you don't know what it means. This may or may not be applicable to a task at hand.

Re:Curl is C by imron on Tuesday March 28, 2017 @12:15AM (#13974023)
> This is in no way just a C thing to deal with all that nonsense. I've not seen any language or library that I'd say does it "well" and saves individual programmers from considering the above.

This is true, however even something as simple as storing the (byte) length as part of the string reduces the complexity and the likelihood for bugs.

Other languages also prevent accidental buffer overruns so while they still need to deal with all the same Unicode problems you mentioned, the program likely won't crash if the programmer gets things wrong. The same is not necessarily true of C.

Re:Curl is C by Manishearth on Monday March 27, 2017 @11:55AM (#13968762)
FWIW in Rust you also tend to avoid allocations, since all string manipulation is done via slices -- safe (ptr, len) pairs. It's pretty neat.

IIRC C++ is getting slices too, so it might be able to get better APIs around string manip. But I've seen decent string manip code that avoided allocations.

Re:Curl is C by burntsushi on Monday March 27, 2017 @09:37PM (#13973309)
> Using a more C way he'd really only have needed a couple of pointers on the stack.

This is pretty much how it's done in Rust too via slices. For example, the standard way to split a string is to create an iterator and it won't do any allocations.

Re:Curl is C by erelde on Monday March 27, 2017 @09:44AM (#13967632)
You can see two of my current school assignement on Github.

I try to avoid manipulating strings as much as possible. ;)

Re:Curl is C by boduh on Monday March 27, 2017 @09:38AM (#13967580)
+1
Re:Curl is C by vbezhenar on Monday March 27, 2017 @10:38AM (#13968059)
A lot of mentioned software was started many years ago, when other languages were hardly viable options. And those programs are good enough now, so there's no much movement to replace them. It's not a good argument for C, IMO. It's like telling that Windows is awesome because so many users use it. But when people started from scratch (mobile world), it turned out, that Windows is not the best OS.

Actually when I'm reading about new software, it's very rare to encounter C. Usually it's something else.