News for hackers. Stuff that matters.
Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @03:52PM (#24642998)
Lisp (Common Lisp) beats all the other dynamic languages by a considerable margin. This is why I am developing Clasp - a Common Lisp implementation based on LLVM that interoperates with C++/C (https://github.com/clasp-developers/clasp.git) for scientific programming.

With Clasp, we get the best of multiple worlds. We get a dynamic language (Common Lisp) with automatic memory management and enormous expressive power that can directly use powerful C and C++ libraries. All three of these languages are "long-lived" languages in that code that was written 10 and 20 years ago still works.

Performance is really important to me and I have written a lot of code over the past four decades. I won't develop meaningful code in any language that falls below Racket in table 4 because these language implementations are too inefficient. I furthermore want to keep using my code over the years and decades and so I won't develop meaningful code in any language where someone else can break my code by changing the standard. My program "leap" was written 27 years ago in C and it is still being used daily by thousands of computational chemists. But it's really hard to improve leap because the code is brittle largely because of malloc/free-style memory management (brrr). For a compiled, high-performance, standard language with proven lasting power and performance - Common Lisp is the best choice.

Re:Energy Efficiency across Programming Languages [pdf] by eikenberry on Wednesday September 30, 2020 @07:17PM (#24645240)
Have you had a look at Pliant? It is basically a Lisp with a modified syntax and has a really nice C FFI implementation. I wrote an Ncurses wrapper for it a decade or two back and remember it being one of the easiest FFI/wrappers I'd ever done.

http://www.fullpliant.org/

Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @09:46PM (#24646372)
No, I just found out about Pliant from your post - but it doesn't make sense for developing large, stable codebases because it's not a standard language. (Sorry Pliant developers - I love your can-do attitude and I'd love to buy you a beer or a coffee sometime and talk about Sisyphean task management.) But Pliant is a reference implementation of a custom language. Programming language design is really, really hard - I wouldn't dare try and so I chose to go with a language that had literally hundreds of person years of design and testing underpinning it. Regarding FFI's - my approach is the same as the very clever pybind11, luabind and the older boost::python libraries. It works by using C++ template programming and letting the C++ compiler do the heavy lifting of generating wrappers at compile time. I recently updated our binding library to use C++17 std::apply with tuples. Freakin' finally! C++ has an apply construct that can be applied to a heterogenous list of objects - wow - lisp has only had it for 50 years! My point is that only recently has C++ developed the introspective capabilities to implement really powerful FFI's. Also - you have to use C++ exception handling for stack unwinding or you will break C++ RAII all the time.
Re:Energy Efficiency across Programming Languages [pdf] by pulse7 on Thursday October 01, 2020 @06:21AM (#24649322)
"because it's not a standard language" Every language was a "non-standard language" at it's beginning... Including C, C++ and Lisp...
Re:Energy Efficiency across Programming Languages [pdf] by TeMPOraL on Thursday October 01, 2020 @01:46PM (#24653970)
What 'drmeister means is that CL as an actual standard, as in ANSI standard. Ditto for C++ (an ISO standard). This means the language has a clear target for implementations to conform to, one that isn't vulnerable to "moving fast and breaking things".
Re:Energy Efficiency across Programming Languages [pdf] by jedbrown on Wednesday September 30, 2020 @04:10PM (#24643220)
Is better C++ interop the motivation for Clasp over Julia (which has lispy roots and CLOS-like open multimethods, though not lispy syntax)? Julia is conspicuously missing in the benchmark, but should do pretty well if you don't include JIT/start-up time.
Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @05:01PM (#24643791)
If the question is why not use Julia? The answer is - for several reasons. I started implementing Clasp before Julia was a thing. The Julia language keeps changing and it doesn't have a standard like Common Lisp, C++ and C do. I need tight C++ interoperation and Clasp does C++ interoperation like no other language I've see. Clasp uses C++ exception handling and calling conventions and this allows us to compile and link C++ code with Common Lisp - it's all LLVM under the hood. Clasp stack frames and C++ stack frames are interleaved and try/catch and RAII and Common Lisp unwind-protect and dynamic variable binding work perfectly between C++ and Common Lisp code.
Re:Energy Efficiency across Programming Languages [pdf] by 7thaccount on Wednesday September 30, 2020 @04:05PM (#24643162)
Didn't you do a talk on YouTube? I recall watching it when you went over the computational requirements of doing something with molecules maybe? I don't work in that field, but remember it being interesting enough to watch the full thing.
Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @04:11PM (#24643238)
I do have a couple of talks on YouTube and thank you (the most recent one: https://www.youtube.com/watch?v=mbdXeRBbgDM). Yes - we are building molecules using our code. I have started a company and we are several months in to developing a new technology for warding off future pandemics (maybe even doing something about the current one). We literally started compiling the first molecules using an application I implemented in Cando (Clasp + computational chemistry) yesterday.
Re:Energy Efficiency across Programming Languages [pdf] by reitzensteinm on Wednesday September 30, 2020 @09:45PM (#24646366)
Your Google talk was one of my top 10 ever, it blew me away.

It's not uncommon in the tech industry to build some mad scientist solution - pardon the expression - because we're trying to do something pedestrian but have painted ourselves in to a corner, eg HipHop VM.

Doing it to help with cutting edge science is genuinely exciting.

https://www.youtube.com/watch?v=0rSMt1pAlbE

Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @11:17PM (#24646953)
Thank you!
Re:Energy Efficiency across Programming Languages [pdf] by 7thaccount on Thursday October 01, 2020 @12:40PM (#24653061)
That is really cool and I feel your pain. I've mostly been dissapointed in numerical computing stacks. You've got C/C++/Fortran that are blazing fast, but very cumbersome to use. They are nice in that you can distribute an executable. Python and Julia are good at finding a sweet spot where your scripting language is fast and user friendly, but distribution is a pain I guess unless you have a SAAS product. Matlab, Mathematica, GNU Octave, Scilab...etc have their own problems like cost or performance and also make distribution painful.

I looked at SBCL, but didn't see good ways to use existing numerical libraries. I'll have to look at Clasp again.

Re:Energy Efficiency across Programming Languages [pdf] by 1980phipsi on Wednesday September 30, 2020 @05:23PM (#24644059)
It would be good to make the slides available.
Re:Energy Efficiency across Programming Languages [pdf] by lliamander on Wednesday September 30, 2020 @04:03PM (#24643133)
Depending upon how you define "dynamic" Java wins by a small margin.

If we treat "dynamic" as a spectrum though, Lisp still gets you a lot more dynamism for a relatively small increase in power consumption (and a somewhat larger gap in performance).

Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @04:14PM (#24643276)
Hey - props to the developers of Java for bringing automatic memory management into the mainstream. But it's not quite what I'm looking for when I want to do exploratory programming.
Re:Energy Efficiency across Programming Languages [pdf] by javajosh on Thursday October 01, 2020 @12:24AM (#24647321)
Indeed, the lack of a real "REPL" experience is quite limiting with the Java language (you can kinda sorta simulate it with bytecode manipulation, but please don't). However, you can get the full REPL experience on the JVM, along with a gc, if you run Clojure! (Which is itself a bit of an oddball language, since it's dynamic code, immutable data Lisp written for a vm that likes static code and mutable data...)
Re:Energy Efficiency across Programming Languages [pdf] by jerven on Thursday October 01, 2020 @06:07AM (#24649251)
Q? did you ever try the jShell that comes with java11+. Curious what a REPL power user thinks about it.
Re:Energy Efficiency across Programming Languages [pdf] by javajosh on Thursday October 01, 2020 @09:47AM (#24650749)
Haven't used it, but I can't imagine it would be great. Java's issue is that you can't do anything outside of a class. So there really is no outside context in which a REPL makes conceptual sense.
Re:Energy Efficiency across Programming Languages [pdf] by jerven on Thursday October 01, 2020 @06:01PM (#24656597)
Well it is not that rigid. You can do code outside of a class/method

  jshell> var x=8
  x ==> 8

  jshell> x*3
  $2 ==> 24
etc.

But I haven't used it either as I never worked with a REPL other than SQL clients.

Re:Energy Efficiency across Programming Languages [pdf] by register on Friday October 02, 2020 @03:33AM (#24660028)
The REPL in CL allows you to redefine running code, even code compiled with high optimization levels. This is a feature of the language rather than a feature of the REPL but it's in the REPL that such features shine. In the best case in Java you can redefine methods when running under a debugger, using a framework such as JRebel or leveraging an application server/framework ( Play for example). When changes are too radical you have to recompile and redeploy. In CL this is only rarely needed. In short a REPL in Java will never be able to reach the level of interactivity provided by a CL REPL simply because interactive development is not built in the language as in the case of CL.
Re:Energy Efficiency across Programming Languages [pdf] by lliamander on Wednesday September 30, 2020 @04:27PM (#24643416)
Agreed - on both fronts.
Re:Energy Efficiency across Programming Languages [pdf] by sk2020 on Wednesday September 30, 2020 @06:18PM (#24644679)
Given that most of the computationally expensive work in Python is usually offloaded to C extensions (and JIT compilation is an option with Python), is the cost savings in electricity that significant? A desktop PC probably only costs about 10 cents an hour to run at most.

Perhaps your interest in CL is because you can recall using it in its prime. Nostalgia Or just novelty is certainly a valid argument.

Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @10:23PM (#24646612)
We are doing computational chemistry, simulating molecular structure and designing molecules, and we want to use thousands of cores and get as much performance as possible. If my AWS bills are any measure - then yes - the cost savings in electricity and computing resources are very significant.

Also, developing and maintaining Python/C++ bindings for complex libraries is very painful and frustrating. I wrote Python bindings for years using boost::python and earlier Swig and keeping bindings working and dealing with the different memory management approaches of Python and C++... bleh - it's a nightmare. At the same time Python changed from version 2 to 3.x and libraries I depended on and my own Python code was being broken and becoming outdated in ways that I had no control over. It was like trying to build a house out of sand.

I've only been using Common Lisp for the past 6 years - after three decades of writing in other languages including Basic, Pascal, Smalltalk, C, Fortran, Python, PHP, Forth, Prolog... Common Lisp feels great, it feels powerful and every function I write I know will compile and run in 20 years. Common Lisp has real macros (programs that write programs! implemented in one language), dynamic variables, generic functions, the Common Lisp Object System, conditions and restarts... There are many features that haven't made it into other languages. Common Lisp makes programming interesting again.

Re:Energy Efficiency across Programming Languages [pdf] by sk2020 on Thursday October 01, 2020 @10:25AM (#24651164)
Would you indulge me with an OOM estimate of your hosting costs for this use case?
Re:Energy Efficiency across Programming Languages [pdf] by Guthur on Wednesday September 30, 2020 @06:52PM (#24644983)
I remember when the OP came into the CL community and it was not long after myself, and we both came long after is hey day.

I dare say that not many in the community are from some bygone time.

The fact is CL has by far and away a more sophisticated runtime environment than the vast majority of dynamic languages, with small talk being a stand out exception.

Some of the language aesthetics have not aged well, not talking parenthesis but rather the hideously long symbol names.

Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @10:39PM (#24646719)
Agreed! There are a couple of really clever xkcd comics about this (google: xkcd lisp). And I also would rather use PATH than default-pathname-defaults and logical pathnames are, uh, crufty. But I see these as minor blemishes on what I think is as close to a perfect programming language as I have seen. Clasp is an implementation of Common Lisp and any differences between what it does and what the standard says is a bug in Clasp that we need to fix. But Cando, Clasp+computational chemistry code, is a superset of Common Lisp and we are adding things to make life a bit more convenient. We even added optional infix arithmetic as a standard part of Cando (I know! I'm going to burn in hell between the ninth nested set of parentheses as a heretic).
Re:Energy Efficiency across Programming Languages [pdf] by scns on Thursday October 01, 2020 @01:00AM (#24647498)
The last sentence painted a big smile on my face
Re:Energy Efficiency across Programming Languages [pdf] by adwn on Thursday October 01, 2020 @01:35AM (#24647696)
> Given that most of the computationally expensive work in Python is usually offloaded to C extensions

That only works if your application's hotspots are in a few, decoupled parts of your code – good examples are FFTs, data compression, or encryption. It doesn't work if you can't cleanly separate your hotspots from the rest of your logic. E.g., if you write a parser and analyzer for a programming language, what part do you want to offload to C? Even if you could identify a small part that takes up the majority of execution time, it would have a complex interface, and it would take a lot of work implementing and testing that interface.

Re:Energy Efficiency across Programming Languages [pdf] by agumonkey on Wednesday September 30, 2020 @03:58PM (#24643086)
I'm often surprised how sbcl is 'benchmark' competitive with other native langs (c, ada, pascal)
Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @04:06PM (#24643169)
SBCL is an amazing implementation and it has an amazing compiler. I would argue that it is one of the best compilers around. Javascript compilers in browsers are pretty impressive - but given the far fewer resources that SBCL development has had, SBCL is remarkable. It's a fast compiler that generates fast code. I would be using it (and do for some applications) if I didn't also need my own large C++ computational chemistry code written over decades.
Re:Energy Efficiency across Programming Languages [pdf] by agumonkey on Wednesday September 30, 2020 @04:12PM (#24643254)
yes sbcl is "only" a tiny impl. Makes me wonder how far franz or other can be .. you know how chez-scheme was leagues above the other implementations while nobody knew until it was open sourced
Re:Energy Efficiency across Programming Languages [pdf] by pfdietz on Wednesday September 30, 2020 @07:52PM (#24645552)
The smallness of SBCL helps speed its continued development. Compiling it from scratch on my laptop and running the smoke tests takes about 1 minute.
Re:Energy Efficiency across Programming Languages [pdf] by 6keZbCECT2uB on Wednesday September 30, 2020 @05:24PM (#24644072)
Would it have been feasible to improve the FFI of existing common lisp implementations? The last talk of yours that I watched showed clasp to be slower than common lisp relative to C.
Re:Energy Efficiency across Programming Languages [pdf] by drmeister on Wednesday September 30, 2020 @05:50PM (#24644399)
There was a time when I asked myself that every day (sigh). But I don't think so. The details involved in interoperating with C++ are so intricate and involved. I don't think I could have gotten to the same place. Meanwhile - we are improving our compiler performance and the performance of generated code. Maybe in the far future I'll start a couple of ancestor simulations and do the experiment...
Re:Energy Efficiency across Programming Languages [pdf] by Const-me on Thursday October 01, 2020 @03:01AM (#24648140)
> The details involved in interoperating with C++ are so intricate and involved.

Yes in general, but you can pick a subset useful enough in practice.

Take a look: https://github.com/Const-me/ComLightInterop

Re:Energy Efficiency across Programming Languages [pdf] by Bancakes on Thursday October 01, 2020 @02:59AM (#24648137)
Why would I use Clasp and not haskell?
Re:Energy Efficiency across Programming Languages [pdf] by higerordermap on Thursday October 01, 2020 @03:44AM (#24648443)
Why would I eat and not drink?