But even here, there is also a spectrum: should the browser engine developer know about the actual pixels of the display, and the "electrocoagulation of Ag nanoparticles between silicon nanostructures that support Mie resonances" [1], and so forth: the layers never stop, hence finally I suppose it is more important actually caring about the product as a whole, in all its effects and side effects, a sort of omoiyari [2], as the Japanese would say.

[1] https://www.degruyter.com/document/doi/10.1515/nanoph-2022-0...

[2] omoiyari: 'give your thoughts and take an action on it', English: https://youtu.be/KJ5kcEQi934?t=71 Japanese: https://www.youtube.com/watch?v=16P_t4ApyhI

Just brainstorming here xD.

[1] https://rustwasm.github.io/wasm-bindgen/examples/dom.html

[2] https://thewagner.net/blog/2019/01/30/load-balancer (off-topic: Rob Pike's Concurrency is not Parallelism https://www.youtube.com/watch?v=oV9rvDllKEg)

A systems program is a background service or a tool that provides the underlying abstractions for applications to run. An application programmer operates with well-defined APIs and, in general, should not be concerned with storage mechanisms, concurrency issues and resource management.

Of course it's fuzzy and unimportant, they're just words for categories we've made up, and didn't even make a very good job of it.

Even if we did make it clear which is which, what's the gain from being able to make a clear distinction? Is it any more productive than fighting over who got the best text editor?

Exactly. I mean, what will you do differently at work tomorrow if your job title is changed from "Systems Programmer" to "Application Programmer"? Approximately nothing.

Is it any more productive than fighting over who got the best text editor?

The text editor discussion is probably more productive!

System programming is about a program being used by another program unlike an application which is being used by a person. A programmer too will not use that program but use an application to develop a program which will then use that program. This makes system programming distinct from application programming in purpose, philosophy and reality.

Contrast this to a GUI program a user interacts with. Its purpose is dependant on its relationship to the user. The philosophy is that the user could click on anything on the screen at any time, so the program had better be ready! The reality is that users live in their own world, free from all concerns of the silicon world, except one - power. They may perceive memory pressure in the form of slowness, but they don't feel it the way the operating system does.

Systems:

- Lacks a GUI.

- There are technical details.

- Doesn't compete with applications.

- Memory pressure is an issue.

Applications:

- Has a GUI.

- Need to be responsive.

- Power usage is important.

- Memory pressure results in slowness.

I don't think we got any closer to a useful answer.

Web browsers are another interesting item that don't fit neatly into this classification either. I mean clearly a browser is something that provides services directly to the user. You use it to navigate the web, view and download documents, store bookmarks, and all sorts of stuff. It's 100% "application level." Except... the browser also provides a runtime, including memory and process management, networking services, graphical rendering, etc. to other software. In fact, a browser is basically a poor man's Operating System. So it's 100% "systems level." Wait, what?!??

The middleware-programming is also systems-programming was also reinforced by Rob Pike in presenting Go as a "systems language" but he now admits[1] it confused people and Go should might be better categorized as a "server" apps language -- i.e. writing Google's backend infrastructure code that doesn't need to be written in C++.

In the 1980s, "systems programming" was usually a synonym for "bare metal programming" because "systems programming" was just a shorter way of saying "_operating_ systems programming". (One writes an operating system to boot up on bare metal.) And systems programming was typically assembly/C/C++ instead of business programming like COBOL or dBASE. In that mode, writing kernel drivers or a UEFI boot loader is also "systems programming". In contrast, "applications programming" was something else.

That said, if today you surveyed 100 random programmers in various domains, I'd have no idea if there's a dominant view of what systems programming is.

[1] from the article: Rob Pike: When we first announced Go, we called it a systems programming language, and I slightly regret that because a lot of people assumed it was an operating systems writing language. What we should have called it is a server writing language, which is what we really thought of it as. Now I understand that what we have is a cloud infrastructure language. Another definition of systems programming is the stuff that runs in the cloud.

I don't think RabbitMQ being written in a "non systems language" is relevant. Its a systems program even if it was written in shell (if it was it might be a bad systems program, but a systems program nonetheless because of it's intended customers and use case).

Take an OS, for instance. You can build on the other parts of the OS (if they exist yet, and/or if your layered architecture permits you to), and the bare metal. That's it. That's all you have.

Contrast that with writing an app on top of, say, a POSIX-compliant OS, and there's a huge difference in how you program. Sure, as other comments point out, there's a continuum of points between the two, but there also is a pretty stark difference between the ends of the spectrum.

Guessing a bit here, but the difference may more generally be that in application programming, you are more constrained by the logic and purpose of the application, and in systems programming, you are more constrained by the system. The app is more stand-alone, and the system is defined by the interaction of the components. From this perspective, it may be fair to think of a microservices architecture as "systems programming".

Database engines are an excellent example of this spectrum. Some simple databases contain no systems programming at all whereas advanced databases are essentially complete operating systems that run in user space. If you walk the code bases along this spectrum, you will see a shift in the construction, character, and design of the code base as more of it transitions from "application" to "systems" code.

The distinction is important because systems programming is a different skillset than applications programming. For example, there is often a requirement to do a lot of sophisticated scheduler design that is integral to the correctness of the software in systems programming. Application programming rarely concerns itself with this, it relies on threads, locks, etc and has the OS sort out the scheduling.

Outside of actual operating systems where systems programming is unavoidable, the primary motivation to do systems programming in a user space application is it can greatly improve performance, scalability, and robustness.

So it really comes down to who your customers are. If your primary customers are other programmers and they are using your program programmatically, then you are most likely doing some kind of systems programming.

That's the only definition that makes sense and maintains a (reasonable if not somewhat arbitrary) distinction between systems and applications once you account for many systems programming tasks no longer being constrained all that severely by hardware, and we recognize that many applications programs have grown and fit the definition of a systems implementation in terms of complexity and number of interacting pieces (another commonly cited definition of systems programming that no longer makes sense. This would necessarily mean that something like Amazon.com is a systems program, which nobody would agree with intuitively).

The value in maintaining the distinction into the present day is mostly in the way it sheds light on various user interface and portability concerns. As in, as a systems programmer must take a very different approach to user experience (since your users are either other programmers of programs) compared to applications programming. A graphical user interface on a web portal is probably not a good idea (at least as the base/primary ui) for a program that is used by other programs and programmers, but it may be a great idea for an application. On the other hand, CLIs make it easy for other programs and programmers to compose functionality with other programs and modules, and so a CLI being the primary interface is a good (but maybe not quite sufficient) indicator that you are probably working with a systems program.

We have the word "middleware" to describe that.

If you look how the word is used, "systems programming" does clearly not mean that (personally, I don't remember ever seeing it used that way). It's more used as an architecture paradigm, where you don't expect a lot of things to be abstracted, for whatever reason. The word choice is obviously because the underlining system that provides those abstractions is architected this way.

The distinction is actually relevant in a lot of contexts. As we get better languages and compilers that provide lower cost or more expressive abstractions, it is getting less and less relevant, but it still has a lot of relevance. And yes, it's a fuzzy concept like any other software architecture one.