News for hackers. Stuff that matters.
Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @02:41AM (#29158874)
The explanation for why it's useful is at the bottom of the page, but it wasn't necessary before glibc got behind and before Linux started adding things that glibc would probably never implement*. Generally, on UNIX-like systems, the C library isn't just the C library but also exposes the UNIX/POSIX API which is just a set of wrappers around calling into the kernel (done with software interrupt 0x80 on x86 Linux.)

* Remember that glibc is part of the GNU project and is meant to be portable beyond Linux, especially to Hurd. It might not make sense for them to implement interfaces that wouldn't work or would require significant work to emulate on Hurd, the BSDs and Solaris.

Re:liblinux: Architecture-independent access to Linux system... by Joker_vD on Tuesday November 09, 2021 @02:52AM (#29158919)
Well, yeah, merging kernel calls into the runtime of one specifically favoured programming language is a silly idea, why not have the kernel calls in a separate, cleanly defined language-agnostic OS-specific library? "It just the way the things has come to work historically"?
Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @03:03AM (#29158968)
Well, first of all the UNIX/POSIX API is a C API, so that question doesn't make much sense. On top of that, there have been no competitive systems programming languages until recently (and certainly none that weren't written in C), so that wouldn't have been worth it. Also, it wasn't really an idea per se: C is just the native language and lingua franca of UNIX and that's just how UNIX developed organically.
Re:liblinux: Architecture-independent access to Linux system... by Joker_vD on Tuesday November 09, 2021 @03:13AM (#29159017)
I would argue that your post is a nice post-rationalization, and is not very accurate historically. The mans in the first versions of UNIX used to show how to do syscalls both in Fortran and C, and only later they've dropped the Fortran because it was too much hassle; and since it was all done way before Ulrich Drepper invented shared libraries, we ended with system calls buried in the C runtime. Also, people wrote complete working OSes in Pascal exactly around the time C was invented, so the part about "no competing systems programming languages" is not true either.
Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @03:42AM (#29159176)
> and is not very accurate historically. The mans in the first versions of UNIX used to show how to do syscalls both in Fortran and C

I don't know what early versions and manuals you mean because that wasn't in v6 or v7. Was there Fortran on UNIX before F77 was introduced in v7?

Re:liblinux: Architecture-independent access to Linux system... by Joker_vD on Tuesday November 09, 2021 @03:59AM (#29159278)
Sorry, I was wrong; for some reason I thought it was Fortran but of course it was PDP-11 assembly. It stayed there up until v7, and was removed in v8. That subtracts from my argument.
Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Tuesday November 09, 2021 @03:09AM (#29159004)
> On top of that, there have been no competing systems programming languages until very recently

There have been plenty. What has changed isn't the lack of competing languages, it's the discontent with C and C++. The Rust hype has coincided with the anti-C and anti-C++ hype and thus the "rewrite it in Rust" movement was born. But Rust wasn't even the first memory safe systems programming language, let alone first alternative to C (and neither was C the first systems programming language itself).

Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @03:44AM (#29159181)
I meant competitive and edited my post now to reflect that and I stand by it. I can only speak for myself with certainty but the benefits never outweighed the costs until more recent languages. If there were really alternatives then more things would have been written in them instead of in C or moving to higher-level languages.
Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Tuesday November 09, 2021 @04:00AM (#29159284)
> I meant competitive and edited my post now to reflect that and I stand by it. I can only speak for myself with certainty but the benefits never outweighed the costs until more recent languages but if there were really alternatives then more things would have been written in them instead of in C or moving to higher-level languages.

You're now conflating "popular" with "better". What makes a language successful isn't always the same as what makes it better (eg there are countless examples of languages failing to gain significant traction (like Zig, Nim, OCaml, D, ) over other languages that had a large company backing it (like Rust did).

Moreover, what makes a language "better" isn't going to be the same for everyone. Some crave simplicity so that the entire language can be memorised (like SPARK) whereas others crave complexity where they can chose the subset of features to support (like C++). Some like C-style braces, some like ALGOL style keywords (like Ada), some prefer S-Expressions (like LISP) and some people even prefer to go direct to assembly.

We've seen operating systems built in Fortran, Pascal, LISP, Oberon and others so it's not even as if I'm making a theoretical point here.

Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @04:14AM (#29159372)
> What makes a language successful isn't always the same as what makes it better.

That's definitely true but I think the odds are that at least one of them would have succeeded if many of them really were competitive. They got a lot of chances.

Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Tuesday November 09, 2021 @04:26AM (#29159435)
You've completely missed the point.

Let's put it in other terms: if your point were true then Javascript wouldn't be the language of browsers. English (which is a mess of a language) wouldn't be the de facto international language.

You cannot conflate "success" with "better". There are so many other factors at play (like money, company politics and which platforms that language is used on happened to luck their way into the mainstream...to name a few).

My point being that the reasons for C's success was as much to do with external factors as it was to do with C itself. In fact in a parallel universe where Bell Labs didn't give UNIX source away and RMS wasn't born (thus wasn't around to create GNU and mandate C as the standard), I'd wager you'd be talking about an ALGOL-derived language (possibly SPARK) as "the best" with C as an academic curiosity; a footnote in history. Or maybe we'd all be writing LISP.

Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @04:47AM (#29159577)
Please stop making accusations. It's obvious that being better is not a sufficient condition for being popular. It's obvious that C and JavaScript are popular now primarily due to entrenchment. I'm not saying that there were no "better" languages by the standards you mean. What I'm saying is that none of them were better enough. They needed to amass such a large number of pros so as to outweigh that one single con of not being the entrenched language for the niche that we're speaking about.

This goes for ISAs too: We use x86 not because it's better or even good. It's because PowerPC (or whatever you prefer) and all those that were better were not better enough to outweigh the massive cost of losing backward compatability, training, habits, relevant capital and all of that. Maybe RISC V, paired with the right microarchitecture, actually does turn out to be so much better that the transition becomes worth that huge cost. We'll see.

We will overcome C, JavaScript, x86 and all that but only once the contenders to displace them are just *that* much better. Many things have had the chance to do that and were better in many ways but not until recently has the balanced tipped, in my opinion.

Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Tuesday November 09, 2021 @05:12AM (#29159744)
> Please stop making accusations. It's obvious that being better is not a sufficient condition for being popular.

I'm not making accusations. You literally said:

> That's definitely true but I think the odds are that at least one of them would have succeeded if many of them really were competitive. They got a lot of chances.

Which is the opposite of the point you're now making. I can only respond to the message you post.

Any way, I think we are largely in agreement even if we're describing the problem from different angles.

Re:liblinux: Architecture-independent access to Linux system... by guerrilla on Tuesday November 09, 2021 @06:32AM (#29160278)
When you say I'm missing the point, confounding things or making the opposite point, you're making accusations.

Both of my points are congruent. Rephrase the quote with "competitive" replaced with "better enough" and this becomes obvious. Your original points were that competitive (i.e. better enough) languages existed and people just got sick of C. I disagreed saying that no, if there had been competitive (i.e. better enough) languages then at least one of them would have succeed by now as they'd all had plenty of chances.

Another argument against this "sick of C" idea is that if that were really true then any of the old languages could have been adopted, now that people are supposedly suddenly sick of C. But that's not what's happening. The new languages are being adopted because they're finally better enough. The old languages aren't being adopted because they were never competitive.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @08:27AM (#29161410)
In the early days of microcomputers the competition was between basic and pascal. Basic was interpreted because decent compilers on that hardware were difficult to achieve, but it was achieved for pascal anyway, so it won. C somehow came later.
Re:liblinux: Architecture-independent access to Linux system... by kazinator on Tuesday November 09, 2021 @07:37PM (#29169582)
Part of that "somehow" may have been Microsoft. They provided a C development environment for MS-DOS, and developed Windows in C, imitating Unix. That probably came in via Xenix. (MS-DOS originally had "Xenix" system calls and features aped after Unix, which tells you that the developers were familiar with Xenix and thus had contact with C and Unix.) Windows API's being expressed in C might have been the catalyst that caused C to explode in the consumer PC scene.

People in the computing field would have known about C in connection with Unix, but Microsoft brought system software written in C to the mass consumer market, smack in the 1980's. That would have raised the profile of C as something proven that could be used to deliver; and you could use the compiler from that same vendor, if you wanted.

Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Tuesday November 09, 2021 @05:53PM (#29168652)
Sorry but you're largely wrong on all counts there.

1. There was a lot more around than just BASIC and Pascal,

2. Pascal wasn't born much before C (roughly a couple of years),

3. and BASIC compilers did exist even as early as in the 60s.

Some examples of other languages (all of which pre-date Pascal and C):

Pascal itself was inspired from ALGOL. So there's a whole class of languages you're excluding. ALGOL was typically compiled too.

Also compiled was FORTRAN. In fact the first FORTRAN compiler was written in the 1957 and the language is credited as being the first ever high level programming language.

Even BASIC wasn't one of a kind. There were a few languages similar (the names of which escape me at this point in time). But BASIC was definitely the most popular of it's ilk.

And lets not forget LISP. It's the second oldest high level language after FORTRAN. LISP has had compilers around since the 60s, although the language is older than that.

Another popular language with compilers back in the 60s was COBOL. And as you maybe aware, some banks are still stuck running COBOL code (albeit modern COBOL looks nothing like 60s COBOL)

And I'm just listing the highlights here. I could also talk about how C evolved from B and B evolved from BCPL. But BCPL's syntax is a lot more like ALGOL than it is C. I could also talk about how FORTRAN, ALGOL and COBOL themselves aren't really a single language but rather a class of languages too. In fact Pascal evolved from ALGOL W and ALGOL W compiler was written in PL/360 which is another ALGOL-like systems language designed specifically for the IBM System/360. Which leads nicely onto the next point that there was also a lot of machine specific or vendor specific languages out there too. In fact it was quite common for organisations to design their own language - possibly even more so then than it is now.

So no, it wasn't just a decision between BASIC and Pascal in the early days. There were a lot of choices even back then. And in fact Pascal wasn't even an option in the early days because it didn't come onto the scene until 1970. Verses the 50s and 60s for all the other languages mentioned above.

Re:liblinux: Architecture-independent access to Linux system... by kazinator on Tuesday November 09, 2021 @07:28PM (#29169510)
It seems you might have paid insufficient attention to the "in the early days of microcomputers ..." qualifying remark in the grandparent comment.

When microcomputers emerged in the 1970's, in particular consumer microcomputers, many languages and other software simply did not migrate to them from Big Iron, mainly for reasons that it just didn't fit.

People who started computing with microcomputers in that era would only have read about Algol or Cobol in Byte or Creative Computing; it wouldn't be found running on their Comodore Pet or what have you.

Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Wednesday November 10, 2021 @03:06AM (#29172254)
Ack yes, I did miss the "micro" part of this sentence. Sorry.

That all said, you'd still be surprised at the number of programming languages that were ported to microcomputers, albeit more so in the early to mid-80s rather than the very start of their evolution in the 70s.

Take the British Amstrad CPC 464, that had compilers for Forth, C and COBOL (to name but a few): https://www.cpcwiki.eu/index.php/Category:Programming_softwa...

I remember seeing a similar array of options for other microcomputers too.

But I'm not disagreeing with the GPs point, those solutions I've exampled were pretty niche. I raise it more for academic interest than as a counterargument.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @03:51AM (#29159230)
Why would it matter what system programming languages are there? Unix can't run other languages?
Re:liblinux: Architecture-independent access to Linux system... by koprulusector on Tuesday November 09, 2021 @05:35AM (#29159897)
The kernel is written in C. Ok, why does that matter? Because its interfaces and architecture are tightly intertwined with C. Remember that C was partially developed to use in building Unix.

So, kernel was written in C and therefore uses C data structures, pointers, and follows C conventions for using them (for example, kernel might return a struct of pointers). This applies to interfaces and structures, making syscalls (depending on which syscall and what is needed), checking return values or error codes.

You don’t NEED this stuff exactly to run on Unix. What and how much you need, or even just convenience, are how it boils down. If you want to use eBPF to trace things from raw network packets, syscalls, file systems, you can do some in Python but you’ll need some degree of C.

You want to run a Django or flask app in just Python? Good to go.

[This article on Linux ABI](https://en.wikipedia.org/wiki/Application_binary_interface) might help explain better/in more detail.

Sorry if this isn’t the best explanation, I’m on mobile and up too late.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @05:54AM (#29160046)
Design of unix was influenced more by PDP-7 than by C. What unix does, it does PDP-7 stuff, not C stuff. C itself isn't limited to POSIX.
Re:liblinux: Architecture-independent access to Linux system... by johannes1234321 on Tuesday November 09, 2021 @08:05AM (#29161185)
Unix can run other languages.

But for answering your question deeper, we have to find an understanding what boundary we're talking about. The kernel syscall interface? Things like Linux's vdso? POSIX library (libc)?

The more we go on in that sequence the more C we get into the interfaces and many of those APIs are essential for writing well-behaving non-trivial programs.

The closer we stay to the kernel's syscall interface the less C you get, but still the syscall interface and C are related. Strings are often NULL-terminated. Arguemtns are passed (via registers) from left to right, not like in Pascal convention right to left. (First arg always in EBX, second in EXC etc.)

A different language will need some interoperability with C conventions for being a good citizen on Unix.

A Pascal operating system for instance would use Pascal strings (historically what tis now called ShortString) where length is encoded in the first byte.

Re:liblinux: Architecture-independent access to Linux system... by Joker_vD on Tuesday November 09, 2021 @08:52AM (#29161698)
> A Pascal operating system for instance would use Pascal strings

Only if the designers would make that decision, and they are not in any way obliged to make such decision! Nobody stops them from using e.g. a moral equivalent of "RECORD size: CARDINAL; str_ptr: ^CHAR END" for passing strings in their system calls (which would probably coincide with using "size_t size, char *str_ptr" in the C prototype at the binary level), or something else entirely.

Is the idea of an OS not exposing its implementation language in its public API that foreign so I have to express it three times in this thread and every time be replied with "No, what are you talking about, of course the OS would use reuse the structures of the runtime of the language that it is written in in its public syscall API"?

Gee, I wonder how the syscall calling convention would look if I wrote an OS entirely in assembly.

Re:liblinux: Architecture-independent access to Linux system... by johannes1234321 on Tuesday November 09, 2021 @09:13AM (#29161961)
It's not what the implementation uses, but "which conventions does the call interface assume" you can use Pascal to create an operating system with a system interface that benefits Fortran. (I know too little about Fortran to describe what that would look like)

With C and Unix the issue of course is that both were developed around the same time by the same people for working together.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @09:27AM (#29162141)
Or it could be designed for interoperability like ODBC which was designed to be compatible with languages that have ISO standards: Ada, Fortran, C, COBOL, Pascal without tightly locking to either.
Re:liblinux: Architecture-independent access to Linux system... by johannes1234321 on Tuesday November 09, 2021 @09:54AM (#29162516)
What is cause, what is effect?

"All" languages (which survived) can call into C.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @12:14PM (#29164369)
C can call into Pascal too. Does it mean libraries should use pascal strings in their interface?
Re:liblinux: Architecture-independent access to Linux system... by johannes1234321 on Tuesday November 09, 2021 @12:50PM (#29164875)
i'm not seeing where i can configure gcc to use pascal calling conventions. looking at https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.h...

And yes, with common Pascal compilers i can write specific functions in a way to provide a C calling interface (called STDCALL) so those functions can be called from C. But for reaching arbitrary Pascal functions I need my own assembly code which passes parameters in the right order and manages the stack "correctly" (in C the caller clears the stack, in Pascal the calleé)

Re:liblinux: Architecture-independent access to Linux system... by Joker_vD on Tuesday November 09, 2021 @10:13PM (#29170645)
...the calling conventions are not the properties of the language itself, or the OS, only of the langauge implementations — the Gnu Pascal Compiler uses the same calling convention as the Gnu C Complier [0], so I think GCC simply decided not to support any other Pascal implementation explicitly and left those to figure out themselves how to link with object files produced by GCC. And yes, generally different implementations of the same language tended to use the same calling conventions to be compatible/interoperable with each other in the first place, although it's not a universal rule, so that's how why x86 calling conventions were labeled with PL names; but that's slight misnomering.

On x64 Linux, everyone managed to agree to use the same calling convention which is of course very convenient. The language of it's written specification is heavily C-biased, but if you squint you can see that it talks mostly about eightbytes and and compound objects and where to shove those so it more or less trivially translates to most other Algol- or C-descendant imperative languages; I believe it can even be reformulated strictly in language-neutral/assembly terms.

On vanilla x86, of course, everyone invented their own calling convention, even different implementations of the same languages: IIRC, Borland C++, MSVC, Watcom, and GCC on Windows all by default produced object files that you could not simply link together without manual tinkering. And (32-bit) Windows, despite being written in C, uses a variation of "Pascal" calling convention in its system API which they called "stdcall". I believe there were some C compilers that actually used "stdcall" by default instead of "cdecl".

Hell, even on x64, Cygwin GCC (on Windows) uses SysV ABI for everything except interfacing with Windows itself where it obviously has to use Windows x64 calling convention.

[0] http://www.gnu-pascal.de/gpc/Importing-Libraries-from-Other-...

Re:liblinux: Architecture-independent access to Linux system... by hnlmorg on Tuesday November 09, 2021 @03:00AM (#29158958)
Because every programming language will have their own opinions on how to link to external libraries and what those external libraries should look like (eg Java libraries are ZIP archives with a .jar extension, JIT languages will have their libraries as raw source code).

But since C is the de facto standard for OS libraries, most languages end up supporting calls to C libraries. Thus C libraries do become the language-agnostic library.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @05:43AM (#29159950)
Most languages can call C libraries because they can call any libraries, not because C libraries are language agnostic.
Re:liblinux: Architecture-independent access to Linux system... by matheusmoreira on Tuesday November 09, 2021 @11:14AM (#29163661)
Higher level languages such as C++ have extremely complex binary interfaces. Only C++ code dares to link against C++ code. Even then there are problems between different compilers and even different versions of the same compiler. New standard version comes out, it requires that standard classes have certain performance characteristics... Suddenly compilers are breaking binary compatibility in order to restablish conformance.

So in order to have usable libraries people will wrap their C++ implementation in C interfaces. Rust faces the same problem. All those nice abstractions result in a complex ABI that will never touched by anything outside the Rust ecosystem. There is no escaping the extern "C".

Some languages are virtualized. Java, C#, Ruby, Python, Javascript all have virtual machines. There's no way to interface directly with code running in those languages. We must interface with the virtual machine instead: import it into our process and ask it to call the functions we want to use.

C binary interfaces are not language agnostic but they are quite simple. The result of compiling a C library is symbols pointing to functions with simple calling conventions. As a result, virtually any language can look up these symbols and call them.

Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @12:24PM (#29164499)
An example of a relatively simple non-C interface is the linux kernel interface. It's so non-C you can't even cleanly interface with it from C.
Re:liblinux: Architecture-independent access to Linux system... by matheusmoreira on Tuesday November 09, 2021 @02:05PM (#29165960)
Yes. The Linux system call calling convention doesn't match the C ones at all. One needs inline assembly in order to interface with the Linux kernel from C.

The ideal solution is to have a system_call keyword built into the compiler that emits the proper instructions for the target platform.

Re:liblinux: Architecture-independent access to Linux system... by couchand on Tuesday November 09, 2021 @08:06AM (#29161206)
That doesn't ring true to me. My FFI experience is somewhat limited (JS to C, JS to C++ via C wrappers, Rust to C, Rust to C++ via C wrappers) but from what I've seen you have to speak C to make FFI.
Re:liblinux: Architecture-independent access to Linux system... by GoblinSlayer on Tuesday November 09, 2021 @10:00AM (#29162594)
Few languages truly speak full C (e.g. through clang plugin), they support ABI, but there are libraries like libpng and gtk with API disconnected from ABI, for them FFI doesn't resemble heavily macro encumbered C API.