kephasp

I've been using Haskell professionally for the last 5 years, I definitely hope I can continue!

How can you prevent code from creating a handle in a new place?

Resumable code is extremely helpful and I'm pretty sure you're describing a chicken and egg issue: if resumable code was readily available in general, people would use it a lot more and it would look like a very normal tool that is often the best.

But when it's only a feature available in niche languages or many languages make it unsafe or unreliable, then very few people will get to the point where it naturally pops up in their mind as a solution to their problem.

This is basically what we already have in Haskell. Debug functions that sidestep the typing system can be annotated with a warning and you can make this one an error while compiling for production.

And in a more general sense, you can ask the compiler to forbid escape hatches altogether.

We already have a pragmatic, reliable solution.

First, you can write a function that's generic on the effects used, so you could write use it in a way that calls an LLM and use it elsewhere in a way that cannot, with the same logic.

But I see the need to change the code as a good thing when it is necessary: I don't want a dependency, inside or outside my codebase, to suddenly add network calls without me knowing it. Same goes for code that was pure and has become stateful. Not knowing this kind of fundamental change is a recipe for a nightmare.

It should be especially apparent with the recent supply chain attacks.

Do you know papers or technical reports that demonstrate the scalability of authorization-preserving search indexes?

I don't doubt they exist but what we hear about are the opposite cases, where this was obviously not implemented and sensitive data was leaked.

One typical way to resolve this is to use Voluntary Oblivious Compliance (VOC). In this capability-based pattern, every storage service could provide an opaque handler to a user that represents their authorization and it can be used to restrict on which documents indexing is done.

http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu...

http://www.skyhunter.com/marcs/ewalnut.html#proofOfPurchase

If the opaque handle is part of the Membrane pattern, you can even avoid most race conditions, because even during the indexing, the capabilities can be used to access documents and that removes the possibility of a TOCTOU race.

http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu...

My team builds frontends in Elm and backends in Haskell and we experience the opposite: FP made the code very easy to reason about and plugging things together is easy too.

John Carmack is already advocating for the use of Haskell for video games.

You need to sketch it and possibly implement it yourself because actual examples don't exist. QED

I read about them a few times, but I never saw them proposed as the main access to recursion. Are there examples of that use?

> So the compiler of your immutable language will still have to insert locks

No it won't, not at the myriads of places they're needed when there are mutations all over the place.

Of course there are mutations in FP, so of course sometimes we need write locks and use things like CAS.

The point is not that FP doesn't use them, but that it uses them far less because they are needed only in a very limited number of places.

When I write code with STM, for example, the only uses of locks or CAS are in the STM engine, not in my Haskell code, and not even in my compiled code, because my code only ever reads memory that's immutable, and immutable data structures.

When I write pure code, there are no locks at all. And that makes it trivial to execute in parallel, either out of order or on several cores.

> Neither C nor C++ have built-in immutable data structures

First of all, yes they do, you can use const on structs and objects and methods to ensure your C/C++ data structure is immutable.

But this is moot anyway:

> so those persistent data structures with their immutable API are implemented using mutable C structs or C++ classes

ZFS isn't something that's in memory, it's a file system… It's a data structure on disk, it's language-agnostic.

> That's because the set of mutable data structures is a strict superset of (i.e., contains all) the set of immutable data structures.

In theory, that's true. It's just a useless fact.

We use immutable data structures because they have benefits. Using the same data structure without immutability may just render it worse as a software tool. So people don't use immutable data structures with the immutability removed.

Immutable data structures are shaped by trade-offs like any other design.

But once it is created, all pieces of code accessing it can be executed in any order because there will never be a write on it again.

Look for actual examples in the wild that weren't error prone and I'd be surprised you'll find any.

I get that you can imagine it. I'm saying it wouldn't work in practice.

You're misunderstanding ZFS' persistent data structure. It's the whole point that it's not doing a copy then modifying something.

Also no one claims there's one immutable data structure that's faster than all mutable data structures. That's a ridiculous strawman argument.

Of course I use combinators more often than I write recursive functions. But if someone finds recursion less readable, I still think they're missing something crucial.

And I don't think recursion is a GOTO that should be avoided. It's a useful tool that's easy to read and reason about.

Experimenting around the state of the art or reimplementing the state of the art ourselves while being informed by it would be fine. I guess it would be a great exercise, as you said.

Do you have examples of this idea that were implemented and weren't insanely error-prone?

A mutable data structure will have such data dependencies all around. An immutable data structure cannot, because it will never be written in after creation.

> It's literally not possible for this to be true

When you need a state that you can rollback, an immutable data structure lets you share the parts of the structure that didn't change between states. That's what ZFS does and that's also how many transactional databases store data.

A mutable data structure would need an entire copy at each state change or it would need to store deltas and apply them backwards when you rollback. Both options are significantly slower than the immutable data structure (where rollback means updating a single pointer).

I wonder if the loss of locality would be compensated by the better concurrency of the overall process…

I'd be interested to see benchmarks.

> Do you mean to say the compiler will insert locks?

I think you're right, it's the compiler that'll insert the locks according to the semantics of the language.

> For an out-of-order superscalar machine, mutability has nothing to do with it;

Yes it does, because it creates data dependencies. If a piece of code B loads the contents of memory where a piece of code A writes that's before B, then you can never execute B before or in parallel with A.

When you use a mutable data structure, and your code is concurrent, you'll force the CPU to put locks everywhere in your code so that several cores won't see stale data in their internal caches.

Even when you aren't writing concurrent code, mutability will prevent the CPU from using its internal concurrency to execute it faster.

Also, immutable data structures can include caches that are extremely simple whereas mutable data structure would face the problem if cache invalidation if they tried.

And a mutable data structure that would operate as if it's immutable? That's a nice recipe for an epic disaster.

Most so-called software engineers fail to do something crucial in engineering: they have zero knowledge about the scientific state of the art about programming.

I agree that the absence of liability enables this.

Do you know an unsafe paradigm that has those advantages over, say, STM or Actors?

But a huge mutable data structure means you can't cache the content between cores of a CPU or between systems in a cluster. So you're describing a scenario where mutability incurs a serious performance penalty.

And why are you saying that requiring more memory would make a program slower? I don't see the link.

Do you have any examples? Tons of system use immutability with extremely high throughputs, most notably WhatsApp.

I don't need infinite memory to make a finite number of modifications to an optimized persistent data structure.

It's not used by many, but it's used by niche environments that have a lot of money, especially in Fintech. When you want your code to be extremely reliable while really fast, there's nothing like languages like OCaml or Haskell.