Thread-safe lock-free sparse integer-to-integer map? No CNC for that.
Translating time between two domains, one of which is linear and monotonic, and the other is non-linear and non-monotonic. No CNC for that.
Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.
I could go on, but you get the point.
What tends to be more like a CNC machine are libraries. For example, realizing that you need some sort of reference-counting system for lifetime management, preferably combined with pointer-like behavior ... and then discovering boost::shared_ptr (later to be std::shared_ptr) ... now that's like getting a new CNC machine. But it doesn't require a new language (and realistically, it didn't even require the library - the library just made it possible to not implement it locally).
I think what I'm really trying to say is that I rarely come across problems where I think that the kind of help offered by the putative "new CNC machine aka new language" is anywhere nearly as substantive as the help offered by an actual CNC machine to a cabinet making company. Put differently, the new tool (language) still leaves the problem essentially as hard as it was before.
p.s. a good friend runs a high end wood shop, and I'm fairly aware of the impact their first CNC machine made to what they could do.
This problem is arguably harder than the example problems you gave and this problem was solved by language primitives that now exist in basically every popular language. These primitives, when used basically change the nature of the language they are used in.
But this only occured because the web was popular and the specific problem of servers and IO changed from a specific problem to a general one. So when someone wants to create a new language it's to attack a general problem.
Your issues in your example look to be somewhat domain specific, so new languages won't really help you in these specific areas you need to handle.
>Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.
I would say that for this example there are enough general issues here that modern languages CAN help you with. For example do you want to program in a language that can guarantee with helpful static error messages that your code will never have a data race or a seg fault or a buffer overflow or a dangling pointer?
Well there's a language that can help you here. In the same vein I've seen languages go even further then this and guarantee that the compiler will never ever let you write code that will make your program crash.
I think we can both agree that these general features that improve safety WILL make the issues you face easier.
The problem was called "the thundering herd": if you had N threads all sleeping/waiting on a condition, and then that condition was raised/signalled, there were no OS primitives available that would wake only a single thread. Instead they all woke up, tried to get whatever work was available, only one succeeded, the rest go back to sleep. Incredible waste of cycles. These days, you can signal a condition in a way that will only wake a single thread that is waiting on it. Problem solved, for every language, without language modifications.
This was NOT fixed at the language level. It was fixed by adding new OS-level primitives that did the right thing.
Async-wait is another wrinkle in this, but for those of us old enough to remember life before pthreads, that was already effectively taken care of using threads (whatever the API) and existing OS-level sleep/wait primitives.
Doing this without threads is popular among the cool kids these days, but that's even harder than doing it with threads. Consequently, various languages have wrapped this sort of code into builtins in the language. Yes, that makes thread-less async wait easier to code, but it doesn't actually address the design problems where you might be using thread-less async wait to accomplish something.
The problem with waveform caching is not data races etc (though of course, those issues are hard enough). It's figuring out what you should cache and when. The best answers vary depending on user behavior, so you need an adaptive approach that isn't particularly linear, and you also a need way to recognize when user behavior means you should clear out everything in the cache and start over.
Re. CnC, how about something like:
@thread-safe @lock-free @sparse
map map[K,V]
(Let's not get hunged-up syntax.)> What tends to be more like a CNC machine are libraries.
If a language or library gives you composable semantics, you may have a programming CnC on your hand. CnC requires minimally one degree of disconnect between the artefact and the artisan. A language compiler/runtime (or library) that applies composable semantics to logical & computational abstractions.
But the syntax is precisely the only thing a language can give you that a library cannot.
The idea that the compiler could somehow pick "the right" implementation of a sparse unordered map based on a list of constraints that combine to create potentially dozens of versions strikes me as far-fetched. Even specifying "lock-free" for example is very, very far from providing enough detail about what is actually required. Wait-free? Readers-not-blocked-by-writers? Writers-not-blocked-by-readers? etc. etc.
I don't dream of a future when compilers (or something) can somehow do all this, and I'm not convinced it will arrive. But I've been (very) wrong before.
This might be true on a similar level to all Turing-complete languages being equivalent, but I’m not sure how useful it is beyond that level. For example, the features that a language provides as directly supported building blocks and the guarantees a language makes about how certain entities will or won’t behave and relate to each other profoundly affects the developer’s experience. That remains true even if some of those features could eventually have been recreated with a library modulo syntax and even if a perfect programmer would always use them properly and never rely on the language to prevent a mistake.
This was fixed at the OS level. But the usage of new system calls involved fundamental shifts in basically every language to account for the new paradigm.
We don't need more programming languages. All languages are either hated or not used. The language isn't the problem and it a poor craftsman that always blames the tool.
I've never been on a retrospective of a failed project and the reason being the language.
I guess I get it though, designing your own language is great fun and I’ve definitely made a couple myself in my spare time.
The hardest problem is usually something else, but IME some languages make it much easier to solve a given hard problem than others. Some languages are qualitatively safer. Some languages are much more expressive, in the sense that I can implement my solution with far less code/time/cost without compromising anywhere else.
There is a difference between whether I can solve a problem and how efficiently and pleasantly I can solve a problem. I find choice of language rarely affects the former but often profoundly affects the latter.
in fact all of the hard parts of modern programming can be characterized by a single problem that all of these languages like go and rust are attempting to solve:
Shared mutating state.
Really the parent and everyone here is just complaining about this one singular problem and how no language has completely solved it yet with a method that's Easy and safe.In fact all these new languages have a sort of similar primary purpose of attempting to abstract this problem away.
We already good methods for dealing with this. In fact, we have several, and what is often the hardest coding question is when to use which one (e.g. RCU versus mutexes).
There are plenty of other "hard" problems in modern programming that have nothing to do with shared mutating state (unless you broaden the definition of that so much that it becomes essentially meaningless).
Here's one example: resolving the tension between cache locality and time-varying data access patterns. Here's another: zero-copy data pathways between user space and hardware. Here's another: resolving the relative costs of fundamental GUI implementation choices (e.g. retained or immediate mode drawing models; pre-rendering components or using vector graphics; absolute positioning vs constraint packing)
Really the parent and everyone here is just complaining about this one singular problem and how no language has completely solved it yet with a method that's Easy and safe.
If my comment was the parent you’re referring to, that isn’t really my position at all.
Certainly there are challenges that come with shared mutable state, but we also have some good tools for dealing with those already. There is a more general problem of managing side effects and external resources, where I think there’s room for better language support, so that might be a better example.
The biggest single frustration I find with many popular programming languages is that once I’ve figured out how to solve the real world problem we’re dealing with, I want to implement that solution as fast as my mind can think and my fingers can type. I don’t want to get bogged down with wrangling general purpose data structures and algorithms, or wondering how to implement an idea with whatever networking or database or mathematical library we use, or writing a screenful of infrastructure plumbing to talk to a cloud service, or trying to figure out all the possible ways that a function I’m calling might tell me it failed and hand-crafting a few lines of boilerplate for every single one.
I haven’t found a language and ecosystem yet that I feel give me that, even though I’ve found many features in reasonably well-known languages and libraries and tools that individually make good progress in that direction.
Some problems require paradigms to solve, and developing bespoke implementations makes maintenance extremely painful.
I would respectfully submit that this is exactly where functional programming shines.
He's doing real-time audio processing. So, first, he's got performance issues. In that environment, it can be very helpful to re-use a buffer without having to allocate one (and without a garbage collector). For most functional languages, that's not a place where they shine.
But, second, he's (probably) got problems like building echo cancellers. The hard problem is, how do you determine whether something is an echo? Writing it in an FP language doesn't help much with that, because the hard part is the definition (and tuning) of the algorithm, not the implementation.
It can be via streaming. The functional portion of the program does all the map and reduce operations on streams. Buffer allocation (and reuse) is automatically handled by the compiler. It's very similar to the way NodeJS hides the event loop. I'm not familiar with audio stuff but I'm sure there's libraries out there that allow you to do extremely efficient audio processing with high level language primitives.
>But, second, he's (probably) got problems like building echo cancellers. The hard problem is, how do you determine whether something is an echo? Writing it in an FP language doesn't help much with that, because the hard part is the definition (and tuning) of the algorithm, not the implementation.
Neural networks can be employed here to "recognize" echos. NNs are basically functional primitives with deterministic inputs and outputs. FP is remarkably appropriate here given that audio is basically a function.
I am extremely familiar with audio stuff, and I can assure you that there are no libraries out there that solve the problems you face when writing a digital audio workstation.
Doing non-realtime processing of file-based data? Of course! Oodles of stuff for that, and you can write it in whatever language you want.
Realtime processing of 240 tracks of audio feeding 64 channels of physical output, with hundreds of plugins? Sorry, but no, you can't do this "with a library" or in "a high level language". This, however, is what DAWs do.
I had assumed that initially you were referring to the difficulty of the “domain problems” when I wrote my comment.
I think these kinds of comments made by FP advocates hurt FP advocacy more than they help, by implying that FP advocates can't see the forest for the trees. We are not in a world where the answer to ever programming problem is "throw FP at it!" We will never be in that world, no matter how awesome FP is, and I'm a huge fanboy.
Go take a look at the Ardour (http://ardour.org/credits) codebase and ask yourself in all seriousness how you could approach a project like that, what the tradeoffs would be, etc. and then ask yourself how serious any of your suggestions sound to someone like who you're responding to.
This by the way is 15 color cameras of extremely high resolution video so it's not trivial. At this level of data you can't expect real time performance without a GPU, so I go straight into CUDA for massive parallel processing performance.
You can think of each pixel in a video as 3 channels of sound, multiply that with HD resolutions and you have thousands of channels of sound flowing through the system at the same time in REAL TIME. 15 * 1080p * 3. Our pipeline also involves real time ML model processing and object tracking all written in C++/cuda.
This of course is across hundreds of similar edge devices all which feed relevant data into the cloud for further processing.
I am not an FP extremist. I work in systems programming which is pretty much the opposite of FP. I am simply stating a specific point here in that I think audio programming for FP can work. I respect the authors comment and I left it at that, even though I'm positive it's possible to write an FP language/framework that handles this specific use case of writing daws. It's basically a massively parallel streaming framework not so dissimilar to what I work on.
The actual processing is easily a functional issue, and the low level memory details can definitely be abstracted away. In fact, it's LARGELY what a daw is when you think people who write plugins.
What really pissed me off about your comment is the superior attitude. Like the "learn how to respect the experts NEWBIE". Seriously I hate shit like that. Seriously who the hell are you?
Anyway ..
> In fact, it's LARGELY what a daw is when you think people who write plugins.
This is the problem though. Yes, that's how people who write plugins think about the host. But it's nothing really like the actual internals.
Comparing video and audio requirements is rarely fair and balanced. I tend to only be interested in realtime processing, not offline. Video involves far more data processing, but also missing deadlines is rarely noticeable by a human (unless you really miss a bunch of them). A single missed audio deadline (even 1 sample) is audible to any human. That's just a reflection of the different ways our eyes and ears work. So it's a bit of complex thing to try to compare the loads.
It's relatively easy to design and implement a parallel streaming framework, especially one without GUI interaction and realtime requirements. Most of the hard stuff comes from the user interaction side and the requirements it imposes on the realtime side. An engine that just takes a set of processing instructions and applies them as fast as possible to as much data as possible is significantly easier in a number of important ways.
FFTW, long the fastest FFT software available, is a program written in OCaml that generates C. You need to think more outside the box!
Another example: Copilot is a Haskell EDSL that generates a hard real-time C program for avionics.
You can write programs functionally and get your imperative performance too.
Indeed. There's a subset of programmers who like to solve "problems" and there's another subset of programmers who like to actualize their intent with minimal exposure to problems.
The later subset is the overwhelming majority both for what programmers themselves want and for what users want out of their programs. Solving "hard" problems is simply a niche desire.
"We don't do hard things because we like doing hard things; we do hard things because we thought they would be easy"
Certainly I naively believed that writing a digital audio workstation would be relatively easy. That was 23 years ago and I'm still writing it.
The language can cause hard problems by itself. Arguably dynamic typing has been partly responsible for driving microservices beyond what's reasonable and thus introduced accidental complexity, largely because they don't have a typing discipline that enforces proper module boundaries.
That's the most obvious example to me, but far from the only one.
Sure, there are situations where I've been grateful for the ability to use Python (because everything is so easy), or Perl (because sometimes its the right tool for the job) or Lisp (because its slippery fluidity just feels right) rather than the C/C++ that I've generally used during that time.
But the hard problems I've faced writing code would (almost?) never be made easier by using functional programming, or Rust, or Go or Swift or Brainfuck. The problems are hard because the problems are hard, and typically for me over the last 23+ years, it has been the combination of generically tough programming problems with performance requirements that ultimately make the choice of language mostly irrelevant.
Sure, someone could offer me a thread-safe sparse integer-to-integer mapping container in another language, but then I have to wonder about what design assumptions were made and how performant it is, and if the language was clearly created to prevent me from ever doing such things myself, then I'm going to be deeply suspicious from the outset that it could possibly have my (coding) interests at heart.
I understand that I'm not the typical programmer these days, and I don't work on entirely typical problems, but I can't help but feel that sometimes the focus on "languages to help programmers" comes from programmers who just don't enjoy their work enough and/or don't have enough to do.