News for hackers. Stuff that matters.
Re:C Strings and my slow descent to madness by 1vuio0pswjnm7 on Thursday April 06, 2023 @05:08PM (#35474472)
"If you are using C and do some non-trivial work with strings you should either use a good library to handle strings or build your own."

   unsigned int str_len(const char *s)
   {
     register const char *t;
   
     t = s;
     for (;;) {
       if (!*t) return t - s; ++t;
       if (!*t) return t - s; ++t;
       if (!*t) return t - s; ++t;
       if (!*t) return t - s; ++t;
     }
   }
I only do "trivial" work with strings but I still use this instead of stdlib strlen. Meanwhile I use software everyday that I know uses POSIX strlen.

NB. I am not the author; this is in the public domain. The author is djb.

Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @05:20PM (#35474644)
I think this naming style should be considered obsolete.

- This function will return the number of bytes, not of characters or codepoints. - str and len are both abbreviations, we should use full words when possible - We can also be more explicit about what the function does, it does not simply returns the string length, it counts characters (or bytes in this case)

Here is how I would name it:

u32 CountBytesInString(char* string); u32 CountCharactersInString(char* string);

And on the implementation side, this work can be done with SIMD instructions, and be really freaking fast, but still, it should be explicit for the user that the work is O(n) complexity, not exactly free.

Re:C Strings and my slow descent to madness by 1vuio0pswjnm7 on Thursday April 06, 2023 @05:29PM (#35474759)
"- str and len are both abbreviations, we should use full words when possible -"

u32 and char are abbreviations.

Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @05:33PM (#35474818)
Yes, this is true. And this is a tradeoff, I think that basic types are so widely used that we can use this style of abbreviation without much ambiguity.

strlen is also pretty unambiguous, but I still have to check what strstr means.

Re:C Strings and my slow descent to madness by lolcatuser on Thursday April 06, 2023 @07:47PM (#35476142)
C isn't Java. Even Niklaus Wirth in Pascal, Oberon, and the like avoided naming their identifiers too long. 'GetStrSz()' is enough to achieve (most of) what you want, assuming certain naming conventions:

- Makes it clear that this returns the number of bytes, assuming a naming convention where `sz` refers to size (in bytes) and `ln` refers to length (in some other unit which would be specified in the type). Note that in C, 'characters' refers to bytes. It's a flaw in how C names its types, yes, but I wouldn't say it should be any different just because other languages do things differently.

- It doesn't use full words because I don't think it needs to. Abbreviations are OK as long as every (invested) party agrees that they're sane, and I think they're pretty sane.

- It makes it explicit that it is performing a calculation (hence, is O(n)) via 'get'.

I don't think all this is necessary, though - I actually think 'strln()' is enough. First, because characters means bytes, I can assume that this function is getting the number of characters (bytes) in a string. I wouldn't expect it to give me anything else! Second, in C, if strings were a struct of some sort, I'd expect to be able to get their length via 'str->ln', which would be O(1). The fact that the length is found through a function in the first place signals to me that it's doing something behind the scenes to figure that out. Remember - that's just my opinion, which I admit is extreme - but I think yours is just as extreme.

Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @08:32PM (#35476522)
Naming is extremely important, and while strlen is a very basic and hardly ambiguous example, consistency is key and I believe that good naming rules should be applied globally or at least at the framework level.

I think that full words and verbs are easier to read and avoid ambiguity.

I guess this is a matter of style and preference.

This anecdote reminds me of the Mutazt type, something I found in a new codebase I was asked to debug. I had to dig for almost an hour to find exactly what this type was.

Turns out it was a char*, a C string. Buried under 4-5 levels of abstractions.

Mutazt = Mutable ASCII Zero Terminal.

Re:C Strings and my slow descent to madness by kelnos on Thursday April 06, 2023 @05:22PM (#35474663)
Out of curiosity, why do you use this? I expect the builtin strlen() to be even more optimized than this. There's a lot more you can do than a simple loop unrolling.
Re:C Strings and my slow descent to madness by YourDadVPN on Thursday April 06, 2023 @05:46PM (#35474977)
Especially when there's a decent chance the compiler would replace the loop with a call to strlen.
Re:C Strings and my slow descent to madness by asveikau on Thursday April 06, 2023 @06:07PM (#35475196)
Don't do this. libc string functions are usually done with hand tweaked assembly or as a builtin by the compiler. They will be faster than this.

This sort of practice is very outdated. The last time it made sense for performance reasons was probably the early 90s or earlier.

Additionally, strlen is not one of the C string functions you want to replace due to defects, the same way you would want to do with crusty old strcpy. If you're working with C strings there is nothing wrong with strlen. (Just don't call it redundantly in a loop body ...)

Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @06:41PM (#35475508)
I've written my own strlen equivalent and benchmarked them against default on different compilers, processors and environments, and they almost always are faster or the same speed.

Default libs are sometimes very optimized but very often they are not, unfortunately.

If you care about performance, you should not rely blindly on the the defaults.

A long time ago I wondered about the performance of memcpy on the Nintendo DS, for sure they would have provided a hand optimized version? And yes, it was handcrafted ARM assembly code, but my own version turned out to be twice as fast.

They simply forgot to use a simple prefetching trick in their implem.

Re:C Strings and my slow descent to madness by asveikau on Thursday April 06, 2023 @07:05PM (#35475740)
Nintendo DS has probably had a lot less scrutiny than a major libc or recent GCC or clang [though you can probably target its ARM processor with that]. Also, for an older embedded platform they may choose to do optimization for code size rather than cycles or clock time.

I'm going to have to doubt the start of your comment. Having seen a lot of libc implementations I think you are better off not wasting time optimizing strlen. Also memcpy, probably memcpy moreso. Most memcpy()s I've seen in the current century are using SIMD instructions and the like. And compilers don't even bother emitting a call to libc for it anymore, they do it as a builtin.

Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @07:40PM (#35476064)
On the contrary, I expected the Nintendo DS SDK to be well optimized, performance of memcpy can be critical on such a constrained hardware. And it was optimized, just not with the best tricks.

I got the prefetching trick from Intel source code, except that I replaced the PLD instruction by a simple dummy load.

And about strlen, you'd be surprised, some implems are very good, and some are not, depends on the compiler and the library. I've ran benchmarks, I was surprised too.

To be honest, I don't really need super fast strlen, but I was curious and also learning to write fast SIMD code, basic string handling is a nice exercise.

Re:C Strings and my slow descent to madness by asveikau on Thursday April 06, 2023 @08:24PM (#35476447)
I think this expectation doesn't vibe with my understanding of how people used to think about embedded or consoles. You shipped them and they were done. The games industry was also often trying to ship quickly. Small teams too. Latest tweaks to memcpy or fine tuning or revisiting the finer points of an already adequate SDK is low priority.

By contrast, many more people are updating optimizations to GCC or clang for arm, more frequently and over a longer timeframe.

Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @09:15PM (#35476839)
You're probably right about consoles, and I was surprised to be wrong, but I checked, just to be sure.

GCC and Clang are very nice compilers, but they are a different thing than the std lib. glibc, musl, the Windows C Runtime, iOS, Android, all have different implementations, sometimes outdated.

Re:C Strings and my slow descent to madness by asveikau on Friday April 07, 2023 @01:19AM (#35478306)
Gcc and clang are relevant here because memcpy is a builtin. It will not call libc for this.

Same is true of MSVC.

It's been that way on most modern compilers for about 20 years.

That's why I'm saying rolling your own may be futile. Compilers, not just libc, have paid a lot of attention to getting those things fast.

Re:C Strings and my slow descent to madness by stephc_int13 on Friday April 07, 2023 @08:20AM (#35480817)
What is true for memcpy (especially on small buffers) is not systematically true for string functions.

But testing is always there to the rescue, and these days we have Godbolt.

Re:C Strings and my slow descent to madness by counttheforks on Thursday April 06, 2023 @09:38PM (#35476998)
Any data to back up that FUD?
Re:C Strings and my slow descent to madness by stephc_int13 on Thursday April 06, 2023 @10:44PM (#35477389)
Of course I have data, do you think I am pulling benchmarks out of a hat?

But I have not published those benchmarks, if this is what you're asking, the Nintendo thing I am afraid I cannot reproduce easily as I no longer have this devkit on hand.

About the strlen benchmark, this is something I've done a few years ago, that could be easy to run again, but I am not sure this is worth the effort just to convince a random dude on the internet...

Re:C Strings and my slow descent to madness by counttheforks on Friday April 07, 2023 @08:30AM (#35480907)
> do you think I am pulling benchmarks out of a hat?

Yes. Share the data or you're spreading FUD.

Re:C Strings and my slow descent to madness by qubidt on Saturday April 08, 2023 @08:47PM (#35499108)
You're being silly. Nothing they said was FUD, it was just a (not particularly controversial) anecdote
Re:C Strings and my slow descent to madness by 1vuio0pswjnm7 on Thursday April 06, 2023 @06:22PM (#35475326)
Here is the musl stdlib strlen. I do not use glibc.

https://git.musl-libc.org/cgit/musl/plain/src/string/strlen....

   #include <string.h>
   #include <stdint.h>
   #include <limits.h>
   
   #define ALIGN (sizeof(size_t))
   #define ONES ((size_t)-1/UCHAR_MAX)
   #define HIGHS (ONES * (UCHAR_MAX/2+1))
   #define HASZERO(x) ((x)-ONES & ~(x) & HIGHS)
   
   size_t strlen(const char *s)
   {
   const char *a = s;
   #ifdef __GNUC__
   typedef size_t __attribute__((__may_alias__)) word;
   const word *w;
   for (; (uintptr_t)s % ALIGN; s++) if (!*s) return s-a;
   for (w = (const void *)s; !HASZERO(*w); w++);
   s = (const void *)w;
   #endif
   for (; *s; s++);
   return s-a;
   }
Re:C Strings and my slow descent to madness by 1vuio0pswjnm7 on Thursday April 06, 2023 @09:28PM (#35476930)
There are HN readers besides myself, not to mention hundreds if not thousands of others, who use various programs that use this function, such as

https://cr.yp.to/ucspi-tcp/install.html

https://cr.yp.to/daemontools/install.html

https://cr.yp.to/djbdns/install.html

https://cr.yp.to/clockspeed.html

https://cr.yp.to/software/qmail-1.03.tar.gz

Re:C Strings and my slow descent to madness by 1vuio0pswjnm7 on Friday April 07, 2023 @06:08PM (#35487639)
glibc strlen:

https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=strin...

   #include <libc-pointer-arith.h>
   #include <string-fzb.h>
   #include <string-fzc.h>
   #include <string-fzi.h>
   #include <string-shift.h>
   #include <string.h>
   
   #ifdef STRLEN
   # define __strlen STRLEN
   #endif
   
   /* Return the length of the null-terminated string STR.  Scan for
      the null terminator quickly by testing four bytes at a time.  */
   size_t
   __strlen (const char *str)
   {
     /* Align pointer to sizeof op_t.  */
     const uintptr_t s_int = (uintptr_t) str;
     const op_t *word_ptr = (const op_t*) PTR_ALIGN_DOWN (str, sizeof (op_t));
   
     op_t word = *word_ptr;
     find_t mask = shift_find (find_zero_all (word), s_int);
     if (mask != 0)
       return index_first (mask);
   
     do
       word = *++word_ptr;
     while (! has_zero (word));
   
     return ((const char *) word_ptr) + index_first_zero (word) - str;
   }
   #ifndef STRLEN
   weak_alias (__strlen, strlen)
   libc_hidden_builtin_def (strlen)
   #endif
   
NetBSD common strlen:

https://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/common/...

   #include <sys/cdefs.h>
   #if defined(LIBC_SCCS) && !defined(lint)
   #if 0
   static char sccsid[] = "@(#)strlen.c8.1 (Berkeley) 6/4/93";
   #else
   __RCSID("$NetBSD: strlen.c,v 1.3 2018/02/06 09:28:48 mrg Exp $");
   #endif
   #endif /* LIBC_SCCS and not lint */
   
   #if !defined(_KERNEL) && !defined(_STANDALONE)
   #include <assert.h>
   #include <string.h>
   #else
   #include <lib/libkern/libkern.h>
   #endif
   
   size_t
   strlen(const char *str)
   {
   const char *s;
   
   for (s = str; *s; ++s)
   continue;
   return(s - str);
   }
Apple strlen:

https://opensource.apple.com/source/Libc/Libc-1244.50.9/stri...

Apple strlen comes from FreeBSD. Until 2009, FreeBSD used an unoptimised strlen.

https://svnweb.FreeBSD.org/base/head/lib/libc/string/strlen....

FreeBSD eventually copied^1 NetBSD's x86_64 strlen.

1. "modeled after", "inspired by" or whatever words the HN commenter prefers to use.

https://svnweb.FreeBSD.org/base?view=revision&revision=18770...

   #include <sys/cdefs.h>
   __FBSDID("$FreeBSD$");
   
   #include <sys/limits.h>
   #include <sys/types.h>
   #include <string.h>
   
   /*
    * Portable strlen() for 32-bit and 64-bit systems.
    *
    * Rationale: it is generally much more efficient to do word length
    * operations and avoid branches on modern computer systems, as
    * compared to byte-length operations with a lot of branches.
    *
    * The expression:
    *
    *      ((x - 0x01....01) & ~x & 0x80....80)
    *
    * would evaluate to a non-zero value iff any of the bytes in the
    * original word is zero.  However, we can further reduce ~1/3 of
    * time if we consider that strlen() usually operate on 7-bit ASCII
    * by employing the following expression, which allows false positive
    * when high bit of 1 and use the tail case to catch these case:
    *
    *      ((x - 0x01....01) & 0x80....80)
    *
    * This is more than 5.2 times as compared to the raw implementation
    * on Intel T7300 under EM64T mode for strings longer than word length.
    */
   
   /* Magic numbers for the algorithm */
   #if LONG_BIT == 32
   static const unsigned long mask01 = 0x01010101;
   static const unsigned long mask80 = 0x80808080;
   #elif LONG_BIT == 64
   static const unsigned long mask01 = 0x0101010101010101;
   static const unsigned long mask80 = 0x8080808080808080;
   #else
   #error Unsupported word size
   #endif
   
   #define LONGPTR_MASK (sizeof(long) - 1)
   
   /*
    * Helper macro to return string length if we caught the zero
    * byte.
    */
   #define testbyte(x)                             \
           do {                                    \
                   if (p[x] == '\0')               \
                       return (p - str + x);       \
           } while (0)
   
   size_t
   strlen(const char *str)
   {
           const char *p;
           const unsigned long *lp;
   
           /* Skip the first few bytes until we have an aligned p */
           for (p = str; (uintptr_t)p & LONGPTR_MASK; p++)
               if (*p == '\0')
                   return (p - str);
   
           /* Scan the rest of the string using word sized operation */
           for (lp = (const unsigned long *)p; ; lp++)
               if ((*lp - mask01) & mask80) {
                   p = (const char *)(lp);
                   testbyte(0);
                   testbyte(1);
                   testbyte(2);
                   testbyte(3);
   #if (LONG_BIT >= 64)
                   testbyte(4);
                   testbyte(5);
                   testbyte(6);
                   testbyte(7);
   #endif
               }
   
           /* NOTREACHED */
           return 0;
   }
Re:C Strings and my slow descent to madness by 1vuio0pswjnm7 on Monday April 10, 2023 @04:38AM (#35510328)
Something worth considering: Sometimes people may use older versions of compilers, e.g., older versions of GCC, for compiling older programs for older hardware. These GCC versions are smaller in size and written to run on less powerful hardware. For example, the gzip'd source tarball for GCC 2.95 from 2001 is 12M while the one for GCC 12.2 from 2022 is 143M, an 11x size increase.