I am sure that inspectability/debuggability are two of the big issues with such an approach. When things fail, the more magic there is, the more difficult it is to fix it. It's even difficult to understand what's happening!
I do think that there is a gap there. You can specify things and have the environment apply your spec. If it's totally naive, like Prolog's "slowsort", it will take exponential time and be impractical for real-world cases. But if strategies can be found and/or devised, and applied, which get some efficiency, it can be practical for many things. When things fall apart, it's clear that powerful debuggers will be needed. A lot of work will be needed on the tools, similar to SQL Query Strategy Inspectors, etc... maybe even professionals specializing just in this. But I think that it will finally allow us to write code in a sane way. Even if armies of professionals have to come after the fact to tweak it so that it's usable, the same as for SQL.
I totally understand what you see as adavantageous in Haskell. I think that same "it's turtles all the way down", where you can inspect and debug every step, is the reason that Haskell cannot gain the power that I am after. Prolog can't either, for other reasons.
I believe a fundamentally different approach is needed. And that's what I'm trying to sketch out.
And thanks to you and many other people hopefully we can learn whether there is something there or there isn't, and if there is, hopefully reach some useful result!
In reality, as you say, you end up with edge cases where you still need to dive into the implementation in order to find out what's happening.
I don't think it's a good enough argument to dismiss the whole paradigm though. Im prefectly happy with my declarative regular expressions that work 90% of the time, and rather take the hit debugging those hairy 10% that end up not working well, than writing my own imperative parser every time and repeatedly deal with those off by one errors etc..
When you say "it never works" I suspect you really mean "it never works in 100% on the cases".
I think declarative is the right default, since it shows the intent over the mechanics, but I deeply distrust any system that won't let me be very specific about the mechanics if I need to be.
It's just as easy to write 'JOIN this billion row table with that billion row table' as it is to forget how hard that is. We may complain that creating indexes is painful, but take some huge complex and slow query and try recreating it in C++ correctly and at least as fast and you quickly enter a world of pain.
PS: It's awesome to be able to write something and then optimize it without worrying that you are going to break something. When it comes to efficiency being able to select performance trade-offs even in an arcane fashion beats testing everything from scratch by a huge margin.
If you can change how you phrase a query and get different performance, while keeping the same semantic meaning, your language isn't just leaking, it's hemorrhaging. It may still be useful, perhaps world-changingly so, but it's a perfect example of the dangers of declarative programming.
As to 'try making it in C++': with a library for doing so with as much maturity as SQL? Sure. It'd probably be reasonably simple, if more verbose. And when it inevitably runs slowly, you can infer the reason for why A runs faster than B relatively simply, because it's doing precisely what you told it to do.
I'd suggest looking at 'declarative' from the point of view of delegation (not in the technical sense, but the usual sense of getting someone else to do a task for you). Ideally you want to specify what you want to be done and you shouldn't care about how it's done. Software is meant to be a labor saving device.
I assume you're ok with the idea of delegating tasks to people. If the software was smart enough I think it'd reasonable to delegate to it as well.
You just shifted the complexity with one sentence, it doesn't make it any easier.
I was pointing out that it's a worthwhile longterm goal, but one that requires sophisticated software to achieve.
We have that. They're called libraries. If I need to sort a collection, I don't write an imperative chunk of code, I just do:
sort(myCollection)
Looks pretty declarative to me.sort(myCollection)
you are still referring to one particular set of instructions. The system parent poster envisions (the way I understand it) would be along those lines:
* User gives SPECIFICATION of sort, along the lines OP writes about in his blog.
* Other contributors provide various implementations which satisfy the specification. Such system would either have to auto-prove that implementation meets specification or just trust contributors that it does.
* Then in your code you can write sort(myCollection) and the system would pick the best implementation.
Erlang, Haskell and Ocaml are examples of declarative languages, and they all have been used to solve very real-sized problems: telecommunication switches, compilers, trading systems, a window manager, etc.
It amazes me how the argument above is still repeated (often followed by "but those systems don't count, give me an example of X".)
I certainly agree with your interpretation more. Typical functional programming isn't really declarative (though, yes, moreso than imperative programs are) and those waters shouldn't be muddied.
1. I encounter a problem; a performance issue or a bug.
2. I can not practically proceed past this point because everything I might need to figure out what is going on has been "helpfully" obscured from me.
Yes, you can still thrash and flail but this hardly constitutes a "fix" to programming. You simply can not help but create an abstraction that not only leaks like a sieve, but is actually multiple leaky sieves layered on top of each other in opaque ways. (And letting us see in is in its own way a failure case too, with these goals.)
Part of what I like about Haskell is that it helps bridge the gap, but doesn't actually go too far. A map call is still ultimately an instruction to the machine. It's not quite the same type of instruction you give in C or C++, what with it being deferred until called for (lazy) etc, but it's still an instruction and it can be followed down to the machine if you really need to without only marginally more work than any other "normal" language. (It may be a bit bizarre to follow it down all the way, but hardly more so than C++ in its own way.)
Contrast this to SQL, which is declarative, and you never have to worry about what the database is doing to answer your question. Except it never works that way and you inevitably must actually sit there and learn how indexes work and how queries are parsed and how the optimizer works to a fairly deep level and then sit there on every interesting query and work out which synonymous query will tickle the optimizer into working properly except that you actually can't do that and you end up having to turn to weird annotated comments in the query specific to your database and then you still end up having to break the query into three pieces and manually gluing them together in the client code.
And I don't even care to guess how many man-millenia have been poured into that declarative language trying to make it go zoom on a subproblem much simpler than general purpose computing. (Well, except isasmuch as they've more or less grown to encompass that over the years, but it's still at least meant to be a query language.)
So, if you think you can fix that problem, have fun and I wish you the very best of luck, no sarcasm. This is the problem I've seen with the previous attempts to go down this route before, and I feed this back in the spirit of helping you refine your thoughts rather than yelling at you to stop.