oddity

It's a fun observation, but the causality in this feels off. There are tons of esoteric programming languages that are difficult to write in and no one talks about because they have no other value. BF is not very useful for writing things in, but it has some value in being easy to implement. Would anyone sane genuinely encourage others to write something nontrivial in BF? What person looks at BF and says "ah, yes, this is a language I might like to write my next spreadsheet app and networking library in, let me play around in it for a bit to see if it might work", much less follow up with "oh no, I got distracted reimplementing it instead"? On the other hand, if I'm implementing an interpreter for fun, I'm going to pick something with a nice spec and my ability to actually use it is irrelevant. BF is great for implementing a spec exactly such that all existing tests and benchmarks just work.

I think the more likely scenario in a counterfactual world where Shen and Forth are difficult to implement is not that we have more things written in Shen and Forth, but that we have strictly fewer reasons to talk about Shen and Forth.

I am not convinced that the group that builds implementations and the group that builds the kinds of things that beget ecosystems for anyone other than compiler developers actually have much overlap. I do, however, think there is a strong overlap between the group that builds implementations and terminal-stage new-language addicts. An ecosystem for other things will maybe happen later if there's some other value to the language.

Where I think the article does hit on something is slightly less direct: users are more tolerant of a messy specification than implementors. C, C++, Python, Java and all the other languages that people actually use are ugly as sin at a semantic level, and yet we get by with copious handwaving. There are many problems, of course, with some of the handwaving users do to reason about these languages (hello, many warts of C), and generally, a preference for spec a user can keep in their head is a big productivity win, but if you want a language that people can use, you can afford to make your language a bit more complicated.

TL;DR: idk, make your pretty language with a beautiful spec if you want. Most languages are unused anyway. But there probably are diminishing usability returns for maximal spec beauty. If you care about that kind of thing, that is.

I think, maybe, the stronger argument is that the area of Generative Programming (which covers just about anything that generates another program) itself is too broad to properly address in a semester course. This course is particularly focused on metaprogramming, where ML has less relevance. This is to be expected, given the instructor.

It's fairly common practice for the title of the course to be far more broad than the topic. See: "Theory of Computing" courses which spend all their time on complexity classes and never mention automata, or graph theory courses which don't ever mention monadic second order logic or spectral graph theory. Decisions have to be made about what to keep and what to cut, at some point.

I spend a lot of time reading papers at the intersection of ML & PL, so I'm a bit sad, personally, but I don't think it's fair to say the course is out of date. Rather, this is just a sign of the known world getting bigger.

There are different forms of argumentation and they have a different tone that maybe doesn't come through so well on the internet without context. I dislike the debate club mentality where arguments are more like a spectator sport than a deliberation most of all, but there are less combatative forms of arguments that are really more like questions with extra scaffolding. I think all people are weakly informed and weakly opinionated about a different subset of most things and it's not always clear when you are in an environment with people who aren't also weakly informed and weakly opinionated. Arguments of the form "I believe this" and "here are my (maybe not so great) reasons" can be a reasonable starting point for laying out your prior knowledge and biases. Making it clear that it is only a weakly held belief is an art form, though, and I can understand why someone might omit that step in an (assumed to be) hostile environment. It's one of the reasons why I think the choice of forum matters. It can be easier to be more open about what you don't know if you aren't subconsciously worried about being attacked for it.

As you said, (I'm paraphrasing) there are often better ways to go about things.

You're not wrong. You can think of internet arguments as a method of developing a thought and collecting counterarguments in a real environment to refine it. The choice of forum affects the quality of the counterargument though, and developing a thought can be done with a crowd more likely to give constructive feedback. If some random website works for you, that's fine.

My statement is more that if you're interested in external impact, exposing someone to a new idea should be treated as the most likely outcome, so assess your effort spent accordingly. I've gotten a lot of value out of reading the different sides of other people's arguments, but I know I personally wouldn't have wanted to spend time being part of it.

I was fortunate to learn this from pointless internet flamewars early in my life. Understand why you're in an argument and what you hope to accomplish by being in an argument. On the internet, it is usually very clear that you will accomplish nothing except maybe introduce receptive onlookers to a new idea, so the choice to minimize engagement is easy.

I think applying this has generally made me more successful outside of the Internet, too, by being more conscious about how I approach conflict. Unfortunately, in the less pseudonymous world where preconceptions and reputation have more weight, the advice also holds, but the calculus is a big mess. Arguments can have only downside risk, but you don't have the option to disengage.

Parallelization and GPUs were the hot story 10-5 years ago, and require(d) a pretty substantial shift in the software stack for less-general gains. You're still hoping the cost-per-transistor goes down. I think recent 400W+ GPUs have shown that we're coming close to the end of this particular S-curve. The big question is whether any of the tricks we have left are broad enough and strong enough to address the economic problem.

I want to believe that there's some world where developers start putting actual craftsmanship into their software (again?), but I think this will only be true for SaaS or related environments where the developer is also the one spending money on the compute. Everyone else will probably outsource the optimization effort to a mixed hardware-software vendor that amortizes the cost (like hyperscale cloud providers) via middleware, instead of just amortizing the cost via hardware.

People (including on HN!) like to say that cycles are cheap but developer time isn't. There's a reason for this that will be invariant of the actual costs: it's the user that pays for the hardware and the developer that pays for development time. End-users are willing to spend money on hardware, but not software, and this has remained true even as companies like Apple and NVIDIA have blurred the boundary. In the current environment of $0 software, how do you (as a developer) fund more efficient software? I think the likely answer will be that developers will happily lock themselves to whatever vendor offers to solve this problem for them. We've seen this in ML already.

I don't usually respond to old comments, so I don't know if you'll read this, but I hope I can encourage you to think more broadly about what "differentiable programming" means.

Different fields have a different perspective on the same set of tools because those tools have different pathological cases in different areas. Context really matters.

> Well, that was my point above: You can't really lump quantum programming together with probabilistic programming, as they are paradigms on different "levels"

This distinction is not useful. If I write in a functional or logic programming language, it gets translated into imperative commands for an underlying architecture that is some mix of dataflow, event driven, automata-based, concurrent, etc... that is then further built on top of some physical atoms where an engineer worried about quantum effects. If I write in a quantum programming language, it will probably go through the same process for at least another 5 years. You might argue that quantum is somehow more dictated by the underlying physical model the way that people argue that imperative programming is closer to the physical world than functional programming. But the "level" doesn't change the usefulness of viewing all of these as "paradigms" worthy of study and analysis on their own terms with their own tools. At the level of studying a programming language, the "level" is a useful thing to be aware of for implementations and motivation but usually not for a theory.

> Even more so, is seems that even defining semantics for differential programming is barely in its starting stages

This is also not a useful distinction. OOP was also, infamously, a point of contention between the academic and outside worlds because it was developed and incredibly prevalent without a rigorous theory abstracting it beyond procedural programming. It became a "paradigm" despite that because there was a set of (informal) tools for reasoning about it on its own terms [1].

Likewise, differentiable programming has largely developed to formalize what makes programs written for machine learning frameworks different from programs written in the imperative/object oriented/functional language they are built on. Autodiff has mostly developed in practical usage, so the use cases are front-running the theory. There's increasingly hardware tailored to the execution model and software developers attempting to program it. There are approaches to problems like discontinuities that people have found solutions for without a rigorous theory justifying their use. There's a structure to why and how people are writing code for these applications as well as an operational theory for how to reason about it, but there's very little compositional, equational theory for these choices.

To most people in machine learning, "differentiable programming" is just autodiff with pretty syntax because the term sprung from attempting to put what they are already trying to accomplish with that implementation on more solid theoretical footing as a computable model of a more general logic. That, hopefully, lets us more efficiently explore what a better domain-specific theory might be and if there are better execution models or logical frameworks. Autodiff itself is increasingly used as an umbrella term for many other methods of differentiation with different edge cases, so this is already happening in practice.

To reduce "differentiable programming" to just its implementation ignores that aspect. It would be equivalent to equating machine learning to matrices. Not unreasonable computationally and not a terrible place to start for a theory, but deeply unsatisfying as a mature domain-specific theory.

The main paper I linked [2] is not about autodiff at all. It's an attempt to establish a connection between differentiation in an analysis sense to models of (not otherwise obviously differentiable) program evaluation. The (unrealized) promise is that the centuries of understanding we have for the calculus of infinitesimals can be applied to the less-mature study of lambda calculus and nondeterministic computation. Papers like [3] cite it because it addresses (discrete) structures that analysis is less interested in and potentially provides a way to connect computation, calculus, and whatever it is that we're doing with machine learning.

> Are you sure about that? I skimmed [1] as I wasn't hadn't read it and it seems to describe a rather restricted set of functions ("types are interpreted as vector spaces and terms as functions defined by power series on these spaces"), as there are many differentiable functions that cannot be defined as power series.

Because it's a PL theory paper, it's not concerned with whether all differentiable functions can be represented, but whether all computable functions can be differentiated. And PL theorists are generally more comfortable than most to accept that most functions cannot be computed and choose a more restrictive model that enables more reasoning power. The category [4] is really the better place to start since it lets us also consider models that aren't vector spaces and [2] is best thought of as a prototype that left many gaps in the theory (for example the requirement on coefficients for convergence is wrong, though I can't remember which paper by Lionel Vaux proved this). It can be thought of as computable in finite instances, but it's unsound even when typed due to the zero term and result of sums.

The quote you cite from [3] is easily misunderstood without that context. As a practice-focused paper, it cares very much about computability. Conditionals and loops are possible in [2] since it allows church numerals and fixed point combinators but it introduces a nondeterministic sum which is exponential in the number of evaluation steps and may diverge (doubly so since it's the untyped lambda calculus...) and is difficult to operationalize. That's what I meant by "wildly uncomputable". So, to them, [2] offers a useful mental framework for higher order features, but is not practical. The theory isn't there yet.

The connections between logic, quantum, probabilistic, and differentiable programming can be understood by how the model treats the exponential modality (!) which converts the otherwise linear term to an analytic one. Differentiation decomposes this to give a sum of linear terms. Differential lambda calculus doesn't put any (more) structure on the sum. Probabilistic programming gives the added structure of a probabilistic sum where coefficients are weights. Quantum programming can be modeled via a Fock space [5] for (!) ([5] predates [2] so is not directly discussed as a differential category here). However, it's unclear what the right model for differentiable programming should be if we want something practical for the resulting derivative (much less antiderivative). Daniel Murfet et al [6] have some related work more directly in the context of machine learning.

[1] https://www.cs.cmu.edu/~aldrich/papers/objects-essay.pdf [2] https://www.sciencedirect.com/science/article/pii/S030439750... [3] https://arxiv.org/abs/1911.04523 [4] https://ncatlab.org/nlab/show/differential%20category [5] https://www.researchgate.net/publication/2351750_Fock_Space_... (sadly a researchgate link) [6] http://therisingsea.org

There's no contradiction: autodiff is a method of implementing differentiable programming. In this example, it is implemented as a type that handles a trace of a program, but everything else is left to the programmer. This is a problem because most of the code I would want to write is not a single trace!

Analogously, I could write a program in C that does message sends and organizes code in a design pattern called "objects" and "classes". Incredibly painful, but workable sometimes. Some people even call it "object oriented C" and go on to create a library to handle it like [1]. Is object orientation not a paradigm because I've implemented a core piece as a library?

No, because that misses the intangible part of what makes a paradigm a paradigm: I structured my code this way, for a reason. In OOP, that reason is the compartmentalization of concerns. The underlying OOP mechanism gives me a way to reason about composition and substitution of components to minimize how much I have to reason about when writing code. Similarly, in differentiable programming, the differentiability of all things gives me a way to reason about the smooth substitution of things because it more easily lets me reason about how the machine writes code.

[1] https://en.wikipedia.org/wiki/GObject

This "discretizes then differentiates" to borrow terminology from [1] which is one of the more accessible presentations and papers. The program might evaluate correctly, but equational reasoning (like you might want for any kind of automated optimizations) is broken. In a toy example like this where you're doing everything manually then you probably don't care, but for larger systems, it gets tiring to do the mental equivalent of assembly programming.

[1] https://people.csail.mit.edu/sbangaru/projects/teg-2021/

The existence of a different kind of CPU isn't a meaningful distinction at the level of discussing paradigms. The semantics are different, so the abstract machine is different. The fact that I need a different set of atoms in my desktop to use it doesn't change the programming language part of the discussion.

The main paper to read is [1] which introduces a syntactic notion of differentiation in the lambda calculus connecting substitution and nondeterministic choice to differentiation in the calculus of infinitesimals sense and also introduces a meaningful notion of Taylor expansion of arbitrary programs. This paper is mostly of academic interest, though. The resulting expansion is wildly uncomputable meaning that more modern, practical papers like [2] cite it wistfully as a dream of what could be achieved. How to computably handle most of the constructs we care about in a general programming sense is very active, open research. At the time the paper was introduced, it was more influential on (and influenced by) work on probabilistic and quantum programming through their related models of linear logic [3]. There are only a few slight axiom differences that separate differential, logic, probabilistic, and quantum programming though, so if you're willing to accept one as a "paradigm", then you should accept the others.

[1] https://www.sciencedirect.com/science/article/pii/S030439750... [2] https://arxiv.org/abs/1911.04523 [3] https://ncatlab.org/nlab/show/differential%20category

I don't need OOP to do a hash table lookup and then an indirect function call with the receiver as the first argument either but that ignores that there's more to a paradigm than the algorithm I use for facilitating it. You can embed unification of expression trees as a library in C++. People implement backtracking all the time in almost every language. Talking about differentiable programming as if it's just autodiff is missing the point of what a programming paradigm is.

There's a mechanism, yes, but that's just a means to an end of efficiently enabling a different way of approaching programming. In the case of differentiable programming, that's continuous code and continuous data enabling program search that doesn't have to use purely discrete methods (like logic programming). If that sounds like autodiff and backprop, then yes, that's because that's a good way to implement it. Tensorflow and PyTorch are DSLs embedded in Python and C++ both useable and used for more than just implementing neural networks, but most people aren't happy calling a library a language until it has a parser and a file extension.

> I'm not sure what you mean here. Could you elaborate?

Most programming languages assume that a variable can only contain one value, or a composite value of values. Differentiable programming lets code be smoothly transformed from one to the other while being meaningful at all points between. In an object oriented case, this would be like having a variable contain an object that behaves like some known object A or object B selectively depending on which choice maximizes the success of the program at any given moment.

Autodiff does not work with for loops or if statements. The current solutions effectively pick a few promising traces through the program and then assume that nothing else exists. To handle it more elegantly (for things like preserving equational reasoning or avoiding exponential blowup) you need to address it at the level of language semantics.

Object oriented programming, for example, doesn't let me have a variable hold half of one object and half of another or let the language derive the code that gave me that object at runtime, but object oriented + differentiable programming does. It's no less of a paradigm than logic, quantum, or probabilistic programming. If you want to, you can view differentiable programming as extending logic programming with a product and chain rule (+ some additional constraints) that allows (smooth, if you want it) interpolation between data and code.

That said, most discussion of differentiable programming is at the level of syntax sugar for reverse mode differentiation, so I can't blame you for that conclusion.

At the time the comment was made the link was https://harvard-iacs.github.io/2019-CS109A/pages/materials.h... where neural networks were mentioned.

See https://news.ycombinator.com/item?id=32295656

The average consumer is more willing to pay for hardware than software, even if all the value is created by the software.

It's the reason why Apple no longer charges for OS updates. Nvidia has essentially the same business model.

It's very hard to sell software to $averageconsumer unless that software is "free"

Apple's ethos of giving more to creators was because they were focused on the niche markets that would spend good money for good products. By the late 90s, creators were their primary users so the survival of Apple meant catering to them even if it meant playing nicely with Microsoft and Adobe. Once they got a whiff of the mass market, it took them until 2019-ish to realize they had lost something.

Apple's modern identity is a lifestyle brand cosplaying as a luxury brand. Creator ("Pro"), to modern Apple, means YouTube, TikTok, podcasts, etc... All the people who might use their very expensive, but not unreasonably so hardware to visibly flaunt their taste over the slightly-less-wealthy Android/Windows plebs. Think high-res-cameras-in-the-iPhone-with-no-high-speed-data-port kind of "Pro" So, they equate Pro to a set of apps and not an ecosystem for enabling those apps, whereas before, they needed that ecosystem for their survival (the Carbon era).

The difference is much more nuanced than this. A modern GPU can (and probably does) do most of what you've listed for a CPU. Speculative execution and branch prediction are a bit less likely to be invested in (because they don't need it as much due to oversubscription), but that's increasingly true for CPUs as well for high-efficiency cores. The difference (at a category vs category level and not specific microarch) is mostly a matter of tuning for particular workloads. I'm increasingly souring on SIMD/SIMT being a useful distinction now that bleeding-edge CPUs are widening in the microarch and bleeding-edge GPUs are getting better at handling thread divergence in the microarch. There is a difference, certainly, but it's difficult to describe in a few bullet points.

GPUs are more likely to have more exotic features than you'll see on a CPU to deal with things like thread coordination and cache coherence, but there's nothing fundamentally stopping CPUs from adding that (or wanting that) as well.

Most people just want an allocator that works reasonably well, and maintaining that expectation means not exposing too many details that you might be held accountable to. If you care deeply, there are usually alternatives.

There's nothing really stopping systems from exposing a hint for controlling this, but usually the people who might care about it don't just want hints, but actual guarantees, and then you have to consider all the users who hint incorrectly (or correctly, but for a different system/version). So, the benefit/cost ratio is low.

Integration of GC with thread scheduling was once an active area of research, but the world has mostly moved on (perhaps prematurely, but so goes).

Depends on how you define innovation.

The hard truth is that there is no free lunch. We like to pretend we're using a Turing machine, but the moment you start caring about performance or memory limits, the abstraction breaks down and you realize that physics dictates that we have a finite state machine instead. This was "solved" about as well as it could be many years ago with however many billions were poured into GC R&D for all the people who don't care deeply about the limits, but nothing is going to magic away that fundamental trade-off unless we discover new physics.

Every innovation since then has been about developing workarounds to deal with that trade-off in more or less sophisticated ways.

C and C++ are some of the few languages where the spec goes out of its way to not depend on an allocator, for good reason, and this is well after you've accounted for the majority of the code that, hopefully, doesn't need to do memory allocation at all. The fact that many programmers don't care is an indication that most code in most C or C++ software is not written with performance in mind. And that's (sometimes) fine. LLVM has a good ecosystem reason to use C++, for example, and it's well known in the compiler space that LLVM is not fast. Less recently, for a long time C and C++ were considered high level languages, meaning lots of software was written in it without consideration of performance. But criticizing the default implementation's performance absent a discussion of its priorities when you have all the power to not be bottlenecked in it anyway is just silly.

It's unreasonable to assume that an stdlib must be designed around performance to any capacity. For most software, the priorities for the stdlib are 1) existing, 2) being bug/vulnerability free, and likely, in the Windows case given Microsoft's tradition, 3) being functionally identical to the version they shipped originally. Linux and macOS have much more flexibility to choose a different set of priorities (the former, through ecosystem competition and the latter through a willingness to break applications and a dependence on malloc for objc), so it's not at all a fair comparison. The fact that malloc doesn't return null all the time is a miracle enough for many embedded platforms, for example, so it's not exclusively a Windows concern. Environments emphasizing security in particular might be even slower.

Multiplication is not a great argument... There's a long history of hardware that doesn't have multipliers. Would I complain about that hardware being bad? No, because I'd take a step back and ask what their priorities were and accept that different hardware has different priorities so I should be prepared to not depend on them. Same thing with standard libraries. You can't always assume the default allocator smiles kindly on your application.

This misses the point of my comment. When you put faith in malloc, you're putting hope in a lot of heuristics that may or may not degenerate for your particular workload. Windows is an outlier with how bad it is, but that should largely be irrelevant because the code should have already been insulated from the system allocator anyway.

An over-dependence on malloc is one of the first places I look when optimizing old C++ codebases, even on Linux and Darwin. Degradation on Linux + macOS is still there, but more insidious because the default is so good that simple apps don't see it.

If you're depending on the performance of malloc, you're either using the language incorrectly or using the wrong language. There is no such thing as a general purpose anything when you care about performance, there's only good enough. If you are 1) determined to stick with malloc and 2) want something predictable and better, then you are necessarily on the market for one of the alternatives to the system malloc anyway.

The semiconductor world is very small by tech standards, and so there are correspondingly many fewer people at the management or upper IC levels that can coordinate large, consequential decisions and even fewer that become well known. Even ignoring attribution biases, it should be unsurprising that there are a few folks that seem to have outsized influence.

But I will say that there are many more people who are not well known outside of the semiconductor world, but are minor legends within it. Unfortunately, the development processes for hardware tend to go through enough hands to strip attribution and most hardware people tend not to talk in public about their accomplishments. Doesn't help that some of the main companies have been stagnant and absorbed in inner and outer turf wars for nontechnical reasons. It's not a great environment for stars to shine.

On top of that, most software people I've seen seem uninterested in understanding and dissecting the hardware enough to appreciate what they've done. If that's the case for software people, I have no hope for anyone else.

The mobile phone market wouldn't have saved them. Qualcomm killed off everyone who wasn't Apple with their licensing schemes. The CPU tech was largely irrelevant because what really mattered was the wireless patents. Being acquired by Apple was probably the best outcome, and we have PA Semi to partially thank for the rise of ARM processors as we consider it today.

Considering this is explicitly part 8 of a series, I would recommend starting with part 1, where this is explained.

https://medium.com/@mario.arias.c/comparing-kotlin-and-golan...

The language website is here: https://monkeylang.org

The PR is undeniably silly, but it's totally plausible that some people just have bad ideas and not bad intent.

The situation doesn't need any more escalation beyond fixing the underlying vector for spam.

Every large org I've ever been at has had someone reply-all to a company-wide mailing list and then promptly spawn a flood of more company-wide reply-alls. Slack's @channel/@here has made this even worse. On a good day, people have a laugh, tell them not to do it again, and we all move on. The only difference here is that it's in public on github.

I suspect there's some psychological phenomenon that convinces people that norms expected of them have somehow been broken because of the chaos.

There's an endless stream of monad tutorials and I think they (almost) all misunderstand the point of confusion. No one gives a f*** about monads, they care about state.

If you've internalized reasoning about a program's execution symbolically and in a time-independent way, then monads solve the problem of how you enforce a correct sequencing of otherwise time-independent operations at a library level without any fancy modifications to the type system. And oh by the way, this structure shows up everywhere and isn't that pretty cool. People who hang around in the formal PL world tend to assume this already because it emerges naturally from how we talk about the semantics of languages via symbolic manipulation.

But if you haven't grasped that yet, then monads solve a problem that you probably don't even realize exists and no amount of rephrasing the monad laws will help you.

I'd almost always recommend anyone new to Haskell ignore monads as much as possible. Fiddle around with evaluating toy functions with pen and paper symbolically to reason about the semantics, and then try to imagine how you could represent the state of a program changing across the page by creating a new object representing the state of the program from the old state. Then, dig in to the RealWorld type and how the IO Monad actually works (not at a type level).