News for hackers. Stuff that matters.
Re:Baby Steps: Slowly Porting Musl to Rust by quotemstr on Sunday June 12, 2016 @01:30PM (#11889193)
I'm disappointed by the idea that something is semantic nonsense, but it's okay to do it anyway. The right line of code isn't "return usize::MAX", but "abort()". (I'd also accept "__builtin_unreachable()".)

In general, if a program enters an impossible state, you need to stop running the program, not do something you know is wrong just so the program keeps working a while longer. That's why strcpy_s is much better than strlcpy.

It's also really not fair to compare the legibility of Rust to the legibility of musl: whatever its other merits, musl is written in that old-school C style that makes one imagine that programs run faster when variable names are shorter and braces are omitted.

Re:Baby Steps: Slowly Porting Musl to Rust by dikaiosune on Sunday June 12, 2016 @01:55PM (#11889312)
This is a hobby project for my own edification, and maybe maybe to eventually show Rust's suitability for this kind of work. While I appreciate that the semantics of returning usize::MAX aren't correct, I'm also not an expert on C standard library implementations. Just a hobbyist writing a little bit about something I'm trying out.

Potentially relevant is this section of the repo's README:

https://github.com/dikaiosune/rusl#goals-and-non-goals

EDIT: After a suggestion in a sibling here, here's a version which neither explicitly aborts (musl doesn't do abort or unreachable), nor uses an explicit return of usize::MAX:

https://is.gd/lck8c7

It doesn't autovectorize, but I suspect that with some work I could maybe convince LLVM to do so.

Re:Baby Steps: Slowly Porting Musl to Rust by briansmith on Sunday June 12, 2016 @04:29PM (#11890086)
IMO, it would be more interesting to see it done top-down, instead of bottom-up. That is, instead of starting with rewriting `strlen`, why not start with rewriting `getaddrinfo`? That would be a very interesting project because you could write `getaddrinfo` with a safe Rust API (i.e. one that doesn't require using `unsafe` to call it), and then wrap it in an `unsafe` function that exports the unsafe C `getaddrinfo` function. This could arguably then be an improvement on the musl code.
Re:Baby Steps: Slowly Porting Musl to Rust by dikaiosune on Sunday June 12, 2016 @04:37PM (#11890117)
I think it very well could be.

However, I would imagine (not having looked at the problem in depth) that one would want to be able to use things like Vec<T> or CString from the Rust standard library when implementing a DNS resolver. Since the Rust standard library depends on libc, I'm not sure how one would properly allow the Rust symbols to replace the linked-in libc symbols while still depending on the rest of libc. Maybe it's not an issue? Maybe there's some magic which could be done?

Nonetheless, I mostly started doing this to see if it could be done, not to prove any point (although I'd love it if a point is proven along the way), so a bottom-up approach with #![no_std] was the easiest way to prevent any sort of cyclic dependencies and/or linker issues.

Re:Baby Steps: Slowly Porting Musl to Rust by briansmith on Sunday June 12, 2016 @04:56PM (#11890194)
> Since the Rust standard library depends on libc, I'm not sure how one would properly allow the Rust symbols to replace the linked-in libc symbols while still depending on the rest of libc. Maybe it's not an issue? Maybe there's some magic which could be done?

Static libraries can have circular dependencies like that. For example, in ring[1] I have C code in one static library, which calls some Rust functions in my Rust code. And, that Rust code calls functions in the C static library. The linker...links them together.

[1] https://github.com/briansmith/ring

Re:Baby Steps: Slowly Porting Musl to Rust by dikaiosune on Sunday June 12, 2016 @05:22PM (#11890314)
Circular dependencies are different than duplicate symbols, though, yes? Anyways, my point is not that it's impossible, but that I'm already working at the edge of my knowledge, and I picked the lowest-risk/smallest-chunk pieces of work to get started.
Re:Baby Steps: Slowly Porting Musl to Rust by spc476 on Sunday June 12, 2016 @08:01PM (#11890961)
On Linux, the symbols in glibc are marked as "weak"---that is, if the symbol "malloc" isn't found in another object file, then the one in glibc is used. It's that way so that users can override system provided functions (non-"weak" symbols that are duplicated are considered an error).
Re:Baby Steps: Slowly Porting Musl to Rust by strcat on Sunday June 12, 2016 @11:37PM (#11891584)
Some symbols are. Not all.
Re:Baby Steps: Slowly Porting Musl to Rust by wildmusings on Monday June 13, 2016 @12:14AM (#11891702)
Yes, but you have to be careful to ensure that no two functions are dependent on each other.
Re:Baby Steps: Slowly Porting Musl to Rust by posborne on Sunday June 12, 2016 @06:59PM (#11890741)
For safe, well-reviewed wrappers around libc (a different problem, but one which is more realistic to solve), I would encourage you to check out the nix crate (I am a maintainer -- the original author was Carl Lerche). It doesn't look like we have a wrapper for getaddrinfo currently, but it is something I think we would welcome.

https://github.com/nix-rust/nix

Re:Baby Steps: Slowly Porting Musl to Rust by quotemstr on Sunday June 12, 2016 @07:17PM (#11890809)
> musl doesn't do abort or unreachable

musl doesn't do dlclose either --- doesn't mean it's a bad idea. Let's not confuse glibc being bad with musl being good.

Re:Baby Steps: Slowly Porting Musl to Rust by dikaiosune on Sunday June 12, 2016 @07:23PM (#11890835)
The goal of the project is to reimplement musl in Rust. It's not a value judgement about musl, glibc, or any alternative implementation. It's about managing the scope of a hobby project the goal of which is to learn more, not to necessarily do any better than those who've come before.
Re:Baby Steps: Slowly Porting Musl to Rust by nwmcsween on Sunday June 12, 2016 @02:21PM (#11889450)
No you don't abort, aborting or trapping is of last effort. A shared library aborting will kill whatever it's linked to which allows zero chance of recovering. There are two sane ways to go about handling 'impossible states': The API is designed in a way that it returns error info somehow, a type system that tracks side effects in order to allow rolling back.
Re:Baby Steps: Slowly Porting Musl to Rust by astrange on Sunday June 12, 2016 @02:54PM (#11889642)
You can't roll back real side effects like network I/O; that's what it means to be a side effect.

Plus crashing (on some OSes) dumps a lot of useful process state and the current stack, but if a function just returns some error code 50 there's not even a way to find which bits of its internals returned it.

If you separate things by process, you can have your crash-only recovery and keep the rest of the system going.

Re:Baby Steps: Slowly Porting Musl to Rust by nwmcsween on Sunday June 12, 2016 @04:37PM (#11890118)
You can roll back most real side effects like network I/O, console output it just depends on details such as protocol and framebuffer type. You also handle errors at the point of issue, not exceptions.

The issue with separating by process is you have to flush the TLB on context switches.

Re:Baby Steps: Slowly Porting Musl to Rust by dleslie on Sunday June 12, 2016 @04:33PM (#11890109)
This is why "interrupted" and "incomplete" are often valid error states.
Re:Baby Steps: Slowly Porting Musl to Rust by CUViper on Sunday June 12, 2016 @01:50PM (#11889285)
Rust does have unreachable!() which panics.
Re:Baby Steps: Slowly Porting Musl to Rust by dikaiosune on Sunday June 12, 2016 @01:56PM (#11889316)
Panics are possible with #![no_std], but it does require that you supply a working implementation of panic_fmt, which I haven't gotten to yet (mostly because I was waiting on panic-as-abort to land which it recently did).
Re:Baby Steps: Slowly Porting Musl to Rust by CUViper on Sunday June 12, 2016 @02:40PM (#11889576)
OK, another option is to make it an explicit infinite "loop{...}" which only exits by returning. The "0.." is effectively infinite anyway, because it will overflow freely in release mode. (But surely you'll hit an invalid memory page before that happens.)
Re:Baby Steps: Slowly Porting Musl to Rust by dikaiosune on Sunday June 12, 2016 @02:44PM (#11889592)
You're right, I think this version is a bit clearer:

https://is.gd/lck8c7

Re:Baby Steps: Slowly Porting Musl to Rust by steveklabnik on Sunday June 12, 2016 @02:02PM (#11889342)
NB: you still need panic_fmt with panic == abort; the default panic handler still prints a message, even though it doesn't go through with the unwinding.
Re:Baby Steps: Slowly Porting Musl to Rust by loeg on Sunday June 12, 2016 @05:24PM (#11890325)
panic_fmt can be as simple as (C pseudocode):

    printf("...");
    kill(getpid(), SIGABRT);
    while(true)
      /* In case kill(ABRT) returns, halt. */;
Re:Baby Steps: Slowly Porting Musl to Rust by _pmf_ on Monday June 13, 2016 @06:07AM (#11892706)
> The right line of code isn't "return usize::MAX", but "abort()".

No, it's not. Assertions and aborts at the library level make a library completely unusable for embedded systems. The caller checks the result for plausibility and then decides to restart or simply not to perform the guarded action.

Re:Baby Steps: Slowly Porting Musl to Rust by tremon on Monday June 13, 2016 @08:23AM (#11893116)
If the caller didn't verify the plausibility of the parameters before calling the function, why is it safe to assume the caller will verify the plausibility of the function result?

Especially for embedded systems, I want the system to blow up on the developer's desk, not in the field, so I'd prefer to fail early.

Re:Baby Steps: Slowly Porting Musl to Rust by nwmcsween on Monday June 13, 2016 @09:02PM (#11899055)
Because the caller shouldn't know specifics on what is valid for the callee unless the caller has a specific and different value of valid. You don't leak information from separate functions this is how you get brittle garbage code.
Re:Baby Steps: Slowly Porting Musl to Rust by programmer_dude on Wednesday June 15, 2016 @08:11AM (#11908775)
> caller shouldn't know specifics on what is valid for the callee

Why? How do you even use an API if you don't know what you can pass into it?

Re:Baby Steps: Slowly Porting Musl to Rust by nwmcsween on Wednesday June 15, 2016 @01:21PM (#11910638)
For example strtol, you may pass whatever (implicitly nul terminated, which is bad) and it either returns a value or an error. I wasnt advocating _you_ don't know what to pass but that the caller doesn't include specifics of the callee.
Re:Baby Steps: Slowly Porting Musl to Rust by rkangel on Monday June 13, 2016 @08:16AM (#11893098)
There's a difference between handling all possible error cases (however unlikely) and handling impossible cases.

The moment you hit the latter case you can no longer make any assumptions about the state of the device and an abort/assert is an entirely appropriate thing to do.

Re:Baby Steps: Slowly Porting Musl to Rust by revelation on Sunday June 12, 2016 @02:25PM (#11889475)
It's really not fair to compare the legibility because that code is not at all doing the same.

He's basically comparing his naive "one byte at a time" strlen to one at least processing at register size. The naive version looks just as good in C, frankly better, because what is this "0.." offset nonsense.

Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Sunday June 12, 2016 @02:46PM (#11889606)
> The naive version looks just as good in C, frankly better, because what is this "0.." offset nonsense.

I'm pretty sure "for i in 0.." is clearer (to a reader unfamiliar with either) than whatever it is you'd do in C. In C it would be something like "for(int i=0;;i++)", right?

Re:Baby Steps: Slowly Porting Musl to Rust by pbsd on Sunday June 12, 2016 @03:08PM (#11889721)
I'm pretty sure a simple while loop is better than either:

  size_t strlen(const char * s) {
    size_t index = 0;
    while(s[index] != 0) {
      index += 1;
    }
    return index;
  }
Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Sunday June 12, 2016 @05:52PM (#11890458)
That's substantially harder to follow than the code in the article - you have to jump around to see where index changes and how that interacts with the loop condition.
Re:Baby Steps: Slowly Porting Musl to Rust by matthewaveryusa on Monday June 13, 2016 @01:06AM (#11891826)
Just out of curiosity how would you rank strlen{1,2,3} in terms of difficulty to read?

  size_t strlen1(const char * s) {
    const char* it =s ;
    while(*it != '\0') {
      ++it;
    }
    return it - s;
  }

  size_t ptr_distance(const void *begin, const void* end) {
    return end - begin;
  }

  size_t strlen2(const char * s) {
    const char* it =s ;
    while(*it != '\0') {
      ++it;
    }
    return ptr_distance(s,it);
  }

  int is_null(const char* p) {
    return *p == '\0';
  }

  const char* find_first(const char* s, int (*predicate)(const char*)) {
    while(1) {
      if(predicate(s)) {
        return s;
      }
      ++s;
    }
  }

  size_t strlen3(const char * s) {
    return ptr_distance(s,find_first(s,is_null));
  }
Re:Baby Steps: Slowly Porting Musl to Rust by kbenson on Monday June 13, 2016 @04:10AM (#11892305)
As someone whos' written almost no C in over a decade, and read minimal C in that time, I would say strlen1 is clearest because it's so simple and succinct, mainly because of how simple the underlying algorithm is.

strlen3 is easiest to read, as you can make fairly valid assumptions about the helper functions, and easily confirm them if needed, and it very clearly breaks down the algorithm into its conceptual components. I think it would also be clearest (and thus winner) if the algorithm was slightly more involved, as the gains in clarity would outweigh the gains of having everything defined in just a few lines.

strlen2 is worst, because the helper function gains you very little for the cognitive overload of having the implementation being somewhere else. You are just as well served in this case by a trailing comment, IMO.

Re:Baby Steps: Slowly Porting Musl to Rust by ThatGeoGuy on Tuesday June 14, 2016 @11:49AM (#11902804)
I agree with this comment more or less, however I would like to point out that strlen{1,2,3} are all wrong, at least in the capacity that they take pointer differences and return a size_t.

In truth, `ptr_distance` and `return it - s;` should both be of type ptrdiff_t, which is not necessarily the same as size_t. In some ways this is why using an explicit index of type size_t that can be incremented is better, because you can avoid some type casting.

Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Monday June 13, 2016 @06:26AM (#11892761)
I would say strlen3 is easiest to read (assuming we're in a codebase that makes widespread use of find_first, ptr_distance and is_null), then strlen1, then strlen2. I think thinking in terms of ptr distances is still needlessly confusing. It's hard to express what we need directly in C (even the Rust version is using nonlocal return which is not the best thing for readability - we could emulate it with goto but that has its own issues) where we don't have generics, anonymous functions, multiple return or tail call elimination. I mean what I'd write in my head (or another language) translates into C as something like:

    typedef struct {
      boolean continue;
      size_t value;
    } result_size_t;

    int indexed_cata_size_t(
        const char *s, void (*operate)(const char, size_t, result_size_t*)) {
      const char *it = s;
      size_t idx = 0;
      // May get initialization syntax slightly wrong but you get the idea
      result_size_t result;
      result.continue = true;
      while(result.continue)
        operate(it++, idx++, &result);
      return result -> value;
    }
    
    void index_of_first_null_u(const char c, size_t idx, result_size_t *result) {
      if(c == '\0') {
        result -> value = idx;
        result -> continue = false;
      }
    }

    size_t strlen4(const char *s) {
      return indexed_cata_size_t(s, index_of_first_null_u);
    }
but I'm not sure that ends up any clearer.
Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Tuesday June 14, 2016 @02:56AM (#11900261)
Nice work, maybe. I really don't know if this is a joke. Wow.

FWIW, strlen in C is 5 to 7 lines of code depending on how you format your code. See my other comment.

For those who are not experienced C programmers, yes the shorter code is more readable.

Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Tuesday June 14, 2016 @04:02AM (#11900501)
result_size_t and indexed_cata_size_t are generic things that would only need to be written once / in the standard library. (perhaps as macros so that they could be generic i.e. not tied to size_t). So the thing to compare is index_of_first_null_u and strlen4 vs alternate implementation, not the whole set of code that I wrote.

And yeah, it's still not nice. In my preferred language it would be something like:

    def strlen(s: String) = s.cata[size_t] {
      case Cons(head, tail) => if('\0' == head) 0 else 1 + tail
    }
(yes, a cons list is not the same as an array and this distinction is important at this level - but the logic should look similar)

I do think there's value in having a clear separation between what kind of looping you're doing and the per-entry logic (index_of_first_null_u). Is it worth the overhead of doing that in C? Probably not for native C programmers, but if I was writing C to be maintained by functional programmers (or me) then I probably would do it.

Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Tuesday June 14, 2016 @05:12AM (#11900740)
I think we would be very unhappy working on the same project. I use functional style in C when it fits... which is rare because most problems are ill-suited to functional programming. The loop on pointer increment and test is completely natural for this problem, as is the pointer math to get the result. This is exactly what the assembly code would do, possibly moving the -1 up to the top and switching from post-increment to pre-increment. (that alternate layout is a C violation that most compilers would accept)

You mentioned that C is missing "generics, anonymous functions, multiple return or tail call elimination". Well, not really.

generics: use the _Generic keyword

anonymous functions: you can autogenerate names via token pasting

multiple return: you can return a struct (living w/o auto-unboxing) or pass a pointer for an extra return value

tail call elimination: there might not be a language requirement for any particular case, but gcc at least is very good about eliminating tail calls -- that said, code which benefits from this optimization is usually hard for humans to deal with

In case you wonder about the C way to count list items...

  if(!node)
          return 0;
  unsigned n = 0;
  do n++;
  while(node=node->next);
  return n;
Like that loop? :-) Curly braces are not needed, and an assignment (plain '=') produces an rvalue that can be tested.
Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Tuesday June 14, 2016 @06:40AM (#11901029)
> The loop on pointer increment and test is completely natural for this problem, as is the pointer math to get the result.

Disagree. "Move the pointer forwards until the value it points to is zero, then measure the distance between where it points and where it started" is a very unnatural way for a human to think about computing the length of a string. The pointer is doing double-duty as a pointer and a counter. It's efficient for the computer, but it's exposing implementation details that don't make a lot of conceptual sense (and don't even match the real implementation these days - modern hardware has to do a lot of work with cache invalidation and the like to maintain the illusion of a unified flat memory space).

> This is exactly what the assembly code would do

Absolutely - but assembly is optimized for the computer, not the human.

> generics: use the _Generic keyword

I don't need to do branching based on a finite set of possible types. I want a generic version of result (where value is of the generic type), and I want indexed_cata to accept an operate that returns whichever generic version of result, and return the same type (I made a mistake in my code - it should return size_t rather than int). Can I do that in C?

> anonymous functions: you can autogenerate names via token pasting

Not good enough I think. I need to be able to create a function at runtime. (I wanted to pass the continuation into operate and then operate can either return directly or call the continuation and return the result of that, rather than have operate return a flag that the caller that indexed_cata checks and loops)

> multiple return: you can return a struct (living w/o auto-unboxing) or pass a pointer for an extra return value

I passed a pointer to result and returned via that for exactly that reason. Could I have had index_of_first_null_u return a result_size_t instead of accepting a result_size_t* and it would all work?

> Like that loop? :-) Curly braces are not needed, and an assignment (plain '=') produces an rvalue that can be tested.

I don't. "=" in a loop test is horrible for readability (it looks like an error (and often is), and has very different semantics from mathematical "=" - pascal-style "while(node := node->next)" would be a much clearer syntax), and returning values that are usually discarded is bad practice (I would like discarding values to be a rare event, probably something that can be a compiler warning - ideally I'd like something like a linear type system). The "do" and "while" look like unrelated statements. The if at the top as the special case is ugly and shouldn't be necessary - when a loop is expressed elegantly the zero case should just fall out naturally.

Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Tuesday June 14, 2016 @04:03PM (#11904897)
Having the pointer "doing double-duty as a pointer and a counter" reduces the potential for bugs. If you do otherwise, then you have more variables to think about and they could get out of sync.

It does match the hardware. There is no cache invalidation.

C programmers commonly think in terms of assembly. They might not envision a particular kind of CPU, but low-level operations are in their mind. The better programmers actually check the assembly sometimes. They know that a "do" loop is normally smaller than a "while" loop because the "while" loop gets converted into a "do" loop inside an "if" that handles the case of no iteration.

The _Generic keyword is resolved by the compiler, so there is no branching. One can also use a macro. Macros can be the normal function-like type of course, but there is so much more. Macros can be used to define functions. Macros can be used with repeated inclusion of the same header file, allowing the creation of many different versions of code. This lets you generate indexed_cata__size_t and indexed_cata__uint64_t and so on, which you would then hook up via a macro called indexed_cata that uses _Generic to dispatch (with zero run-time overhead) to the correct implementation.

I probably don't really get what you're looking for with the continuation stuff. Wikipedia compares it to setjmp/longjmp and goto, which doesn't sound too nice. This is all completely off-the-wall alien to a C programmer. It's hard to get excited about learning to understand something that is associated with really slow programming languages.

Being "able to create a function at runtime" is normally incompatible with having a small high-performance runtime. In C you have a choice: fake it, or do things the awesome way. You can fake it with macros, possibly including ones that expand out to define functions. The awesome way is to JIT, and yes I've done this: allocate a chunk of memory, write out some binary instruction code, deal with any permissions/cache/pipeline issues, and then call it as a function or use a bit of inline assembly to reach it. Oh, there is another way that might be more your style: create C code on-the-fly, feed it into the compiler to generate a library, dlopen the library, and call your new function.

Yes, you can return a struct. What you don't yet get is any syntax to automatically split that apart in the caller. The caller will end up with a struct. Hopefully the next C revision will improve this.

Expressing my loop in an alternate way is likely to generate code that is bigger and slower. This particular case is trivial enough that gcc 4.8.4 can optimize it down to the exact same bytes, and a newer gcc probably just substitutes a call to strcpy unless you pass -ffreestanding to stop that optimization. Normally it makes a difference.

Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Tuesday June 14, 2016 @05:10PM (#11905341)
> Having the pointer "doing double-duty as a pointer and a counter" reduces the potential for bugs. If you do otherwise, then you have more variables to think about and they could get out of sync.

If you're just adding 1 to both each time then it's pretty hard to get that wrong. I think storing the original pointer and subtracting at the end has more potential to go wrong.

> I probably don't really get what you're looking for with the continuation stuff. Wikipedia compares it to setjmp/longjmp and goto, which doesn't sound too nice.

It's not well-explained - the terminology is arcane, people use the word to mean different things. But done right it can give you the performance and flexibility of goto but with the clarity of regular function calls, and all your control constructs can just be library functions.

> It's hard to get excited about learning to understand something that is associated with really slow programming languages.

Shrug. Almost all programming languages are fast enough these days. Correctness is often much more of an issue.

> Being "able to create a function at runtime" is normally incompatible with having a small high-performance runtime. In C you have a choice: fake it, or do things the awesome way. You can fake it with macros, possibly including ones that expand out to define functions. The awesome way is to JIT, and yes I've done this: allocate a chunk of memory, write out some binary instruction code, deal with any permissions/cache/pipeline issues, and then call it as a function or use a bit of inline assembly to reach it. Oh, there is another way that might be more your style: create C code on-the-fly, feed it into the compiler to generate a library, dlopen the library, and call your new function.

I don't need anything that complicated. Just the ability to partially apply a function would do. Or even ordinary classes. Which I mean sure, I can emulate by passing a struct consisting of a function pointer and the first argument as data, but that's going to be awful to read.

Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Wednesday June 15, 2016 @11:54AM (#11910076)
Functional programming throws away the "clarity of regular function calls" by abusing them or their syntax for control flow. It's an unreadable mess. Having control constructs be library functions seems to be an extension of this, making the situation much worse.

I write code to emulate the modern multi-GHz PC and other things running at about a GHz. It's timing sensitive, so the emulator needs to keep up, and we really want to emulate SMP on one core. My coworkers write code to treat a binary executable as a giant equation, solving it to find inputs which cause crashes. We pay lots of money for giant rooms full of computers running our code.

All programming languages are way too slow.

I'd switch to FORTRAN if it was 2x faster. I'm using C99 with lots of non-standard extensions and inline assembly. We use SSE intrinsics to vectorize things by hand.

Meanwhile, people like you are making my web browser too damn slow. It's 2016, and it feels like 1996. Other than a slight increase in pixels and always getting 24-bit color, there has been no improvement in speed despite the hardware being something like 1000 times faster. WTF. I paid good money for that hardware because I wanted things faster. I didn't buy that hardware to be wasted. Your code is not special. Each programmer tests his code in isolation, giving it the whole machine, but out in the wild it must share the machine with other code and/or with numerous instances of itself.

Sometimes gcc's optimizer will partially apply a function. I've seen the result via disassembler. This beats partially applying functions by hand. For example, the function may do different things if a certain parameter is NULL. The compiler makes a version of the function for when the parameter is NULL, and another for when it is not.

Passing a struct with a function pointer and argument seems a little annoying for a trivial situation, but it makes lots of sense when there are multiple functions and multiple arguments. For the trivial case, one would normally just pass the function pointer and argument separately.

I think I've written code that matches your "partially apply a function" idea pretty well, using macros. Like this:

There exist functions for reading/writing memory with names like read_mem_8 and read_mem_32, for several different sizes. I want to make wrapper functions called read8 and read32 and so on. The wrapper functions check for alignment, check a TLB, check for breakpoints, and then call the wrapped function or do the access directly. I make a macro that expands out to the function I want whenever I "call" the macro. No, I don't mean a regular function-like macro. Mine actually creates a real function. I invoke the macro itself at top level, outside of any function, right after creating the macro. I invoke it for each size that I want. Macro expansion generates a function that I can later call.

Invoking it looks like so:

  define_read(8)
  define_read(16)
  define_read(32)
  define_read(64)
  define_write(8)
  define_write(16)
  define_write(32)
  define_write(64)
Note the lack of a semicolon. These macros expand to actual functions that I can call. I could even take the address of the resulting functions if I wanted to, not that I do.

Invoking the result of the macro expansion looks like so:

  write32(cpu,addr,val);
For bigger things, one would put the template-like version of the function in a .h file and then include it repeatedly, each time with different choices. Like so:

  #define CHOSEN_BIT_WIDTH 32
  #define CHOSEN_ENDIANNESS ENDIAN_BIG
  #include "def_stuff.h"
  #undef CHOSEN_BIT_WIDTH
  #undef CHOSEN_ENDIANNESS
  
  // ... and then again with different choices
Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Thursday June 16, 2016 @04:37AM (#11914793)
> Functional programming throws away the "clarity of regular function calls" by abusing them or their syntax for control flow. It's an unreadable mess. Having control constructs be library functions seems to be an extension of this, making the situation much worse.

Shrug - that's the opposite of my experience. When you're working in a world where functions are functions, you don't need explicit control flow - you just want to express what the return value needs to be, and have the computer do whatever it needs to to make it happen as quickly as possible. Restricting yourself to a call stack model is something you have to do for sanity in a language with pervasive mutability and side effects, but as a C programmer you surely know that it means giving up some performance. In a language where your functions are pure (which is often where you want to be for optimization anyway - compare GCC's SSA form) a lot of the need to explicitly reason about control flow goes away.

> Sometimes gcc's optimizer will partially apply a function. I've seen the result via disassembler. This beats partially applying functions by hand. For example, the function may do different things if a certain parameter is NULL. The compiler makes a version of the function for when the parameter is NULL, and another for when it is not.

I'm asking to express a partially applied function in code, not for performance but for clarity and readability.

> I think I've written code that matches your "partially apply a function" idea pretty well, using macros. Like this:

I don't want to do it at compile time, I want to do it at runtime. A super-trivial example would be something like:

    def add(x: Int, y: Int): Int = x + y
    def addX(x: Int): Int => Int = add(x, _)
i.e. I want addX(5) to return a (pointer to a) function that takes an int and returns an int that's that int + 5. How do I do that in C?

    int add(int x, int y) { return x + y; }
    int (*)(int) addX(int x) ???
I can return a struct that contains 5 and a pointer to add, but that requires whatever calls addX to know about the difference between this and a normal function.
Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Friday June 17, 2016 @11:19PM (#11926887)
Most programs aren't about a return value. They are about global state, including IO and huge data structures. The functional programming hacks for IO are gross. The issue of dealing with huge data structures is generally not solved; when you make a tiny change to a terabyte you have an awkward problem.

It's not standard C, but gcc will let you mark a function as being pure. Even without that, the compiler may notice.

I don't agree that expressing a partially applied function at runtime helps clarity and readability. I suppose tastes differ, and some people have bad taste. :-) The real trouble is that this is not something that is natively supported by the hardware. Your language either interprets a very unnatural machine, or it drags along a compiler.

Interpreting a very unnatural machine is unappealing. It distances you from reality, making it difficult to reason about how your code will perform on the hardware.

Dragging along a compiler is sometimes, rarely, justified. It's appropriate when the partially applied function must run for a long amount of time. Sometimes people even drag along a compiler for a different processor, such as a GPU. We find this to some extent with CUDA and OpenCL.

Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Monday June 20, 2016 @11:17AM (#11938560)
> Most programs aren't about a return value. They are about global state, including IO and huge data structures.

Not my experience at all. Most of the time if you're using a computer it's because you want to, well, compute. I/O happens but very often it's a simple matter that can be pushed to the edges, and the business logic that you actually need to be thinking about can be cleanly encapsulated.

> The functional programming hacks for IO are gross.

Only because I/O is gross. I find it's more a case of: functional style forces you to expose the ugliness that was always there underneath. Often you uncover things like the family of security vulnerabilities where an application creates a temporary file and then opens it and an attacker can replace it with a symlink in between. If you really want to say "do these I/O operations whenever, it'll all work out" then it's trivial to do that, even in Haskell (you just add "unsafePerformIO $" before all your I/O calls), but most people find it's actually worth explicitly figuring out which order things are supposed to happen in and doing that.

> I don't agree that expressing a partially applied function at runtime helps clarity and readability. I suppose tastes differ, and some people have bad taste. :-) The real trouble is that this is not something that is natively supported by the hardware. Your language either interprets a very unnatural machine, or it drags along a compiler.

Not at all. Every language with classes manages to do this (even Java can do it nowadays), C++ included. It's not hard to implement. What it does require is one more layer of indirection around function calls (if you're really bothered about the overhead you can make it opt-in i.e. "virtual" in C++). I could write my own code to do this in C, but a) I couldn't use the "function(arg1, arg2)" syntax because there's no way to override that (would have to do something like a macro, "VIRTUAL_CALL(function, arg1, arg2)") and b) it wouldn't interoperate with any of the rest of the ecosystem. You might not want to use it in every program, but this is the kind of thing that absolutely needs to be part of a language-wide standard for programs that do want to use it, so that libraries from different people can interoperate with each other.

(FWIW I think the idea of corresponding to the hardware is a false idol - or at least, if there is a way to achieve it, C isn't that. GCC generates something very different from a hand translation into assembly. Even assembly no longer corresponds to what the actual processor will do (microops). Modern processors are inherently parallel and no remotely mainstream language expresses that accurately.)

Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Saturday June 25, 2016 @01:25AM (#11975351)
Probably you'd end up with this:

foo->bar(foo,arg1,arg2)

It's not as simple-looking, but that isn't all bad. It's very clear what is going on. The overhead isn't hidden.

Things needing call-back functions need to support an extra void* of course, and sadly it is often left out.

Understanding how compilers translate C into assembly is a skill that must be practiced. I suggest getting in the habit of looking at the binary. After a while, you will know what to expect. The optimizers are not magic. They do fairly predictable transformations on the code. You can get a feel for when hand-optimization matters, and when it is just pointless. An awareness of cache lines and TLB entries is helpful too.

Re:Baby Steps: Slowly Porting Musl to Rust by lmm on Monday June 27, 2016 @09:15AM (#11985937)
> Things needing call-back functions need to support an extra void* of course, and sadly it is often left out.

Well when doing the right thing means writing more verbose code, it's no surprise that people do the quick-and-dirty thing (see http://www.haskellforall.com/2016/04/worst-practices-should-... ). Since "everyone knows" that best practice for a callback is to take a function pointer and a void * , wouldn't it be better to have that idiom built into the language? (i.e. a concise syntax for the pair of callback-and-void * , and a concise syntax for "invoking" the pair, passing the extra void * along with the "ordinary" arguments).

Re:Baby Steps: Slowly Porting Musl to Rust by burfog on Tuesday June 14, 2016 @02:43AM (#11900211)
Here is the way that is correct and obvious to a modern C programmer. Real C libraries use assembly and/or multi-byte methods that violate aliasing rules.

  size_t strlen(const char *restrict const s){
          const char *restrict t = s;
          while(*t++)
                  ;
          return t-s-1;
  }