Anyway, I think there is a fundamental design difference between Common Lisp and other 'first class' programming languages: Common Lisp was designed as a way to use computers because a computer user would sit down at a Lisp Machine (which was the future when Common Lisp was designed) and use Common Lisp to do ordinary computer stuff like store their recipes and manage appointments and copy files between directories. It's reflected in the :cl-user package not being called :cl-programmer and the logic of typing (in-package :tps-report) on Saturday at the usual time.
Most other languages don't have this idea...In Unix users don't fool around with stdio.h. In Python, there's no clean way of switching between applications at the REPL -- or rather interpreter -- because of how Python handles the hard job of naming things (like most languages, there's a bit of punting on third down with the catchall epithet 'unpythonic').
This makes it hard to get one's head around Common Lisp, but it explains why a person might leave a REPL running for a week and make better progress because of it.
Common Lisp was actually designed so that you don't need a Lisp Machine. The people working on it were from CMU (Unix workstations), Lucid (Unix workstations), Franz (Unix/Windows), Apple (Mac), Symbolics (Lispm, PC), Xerox (Lispm, Unix), and many others.
Users used an editor-based IDE (like Franz ELI with GNU Emacs, ILISP with Gnu Emacs), a special IDE usually written in Lisp (Franz, LispWorks, Macintosh Common Lisp, Golden CL on Windows, ...) or even a Lisp Machine which combines the IDE with the operating system.
> copy files between directories
The Symbolics Lisp Listener (REPL + commands) has a command language/interface with a lot of comfort - but with its own usability problems.
You would often type
Copy File *.lisp.newest >subdir>
instead of using the Lisp function COPY-FILE.Still today I tend to keep the Lisp IDE running for days/weeks/months... as long as possible. Sometimes Linux tells me I have to restart the machine after some software update...
As a point of contrast, Smalltalk was developed in part with the idea of Dynabooks in the hands of children. Hence its stereotypical use cases presume a substantial difference between the cognitive capabilities of the system's users and the cognitive capabilities of system's programmers.
Common Lisp was originally mostly a simplified and modernised version of Lisp Machine Lisp. It was supposed to be 'cheaper' for users (industry, military, ...), when delivering software. Without the 'hardware dongle' of a Lisp Machine, which would cost a lot - both hardware and software.
One of the main purposes of its existence was to be a standard Lisp able to work on a wide variety of hardware - hardware which was less powerful. Thus the design in the early days was also driven by taking features away from what a Lisp Machine would provide. Less features, easier to implement, able to integrate into different environments.
For example Common Lisp provides type declarations. This was added for non-Lisp-Machines. The Lisp Machine compiler ignored type declarations. It did not use it. The CPU does runtime type checking/dispatching on a Lisp Machine. Always.
Everything fancy, which was difficult to implement on small machines, was removed. The result was CLtL1.
The assumption was that you could develop a Lisp based application (say, an expert system helping with jet turbine maintenance) on some platform of choice and then deliver it to the Airforce on some rugged PC, where it would be used in some airbase. People then thought, why not develop on the PC? Thus various implementations for PCs came up.
NASA was putting it on some embedded computer running million miles away from earth, controlling a spacecraft. There was really no limit to where one would have wanted it to be deployed... thus it ended up controlling your cleaning robot (Roomba)...
The idea that user = programmer was part of the MIT AI Lab culture before there were Lisp Machines. For example, the top level of ITS, the PDP-10 OS they used, was the debugger. Imagine if the default Linux shell was GDB!
Less obvious things were having a variety of filesystem-related functionality as built-in statements with special syntax, rather than functions. For example, renaming a file: NAME "foo" AS "bar".
You also have a third option: you can use Lisp as a sort of interactive command-line calculator-plus-kitchen-sink. Leave it running and teach it to do odd jobs for you. If you accumulate your odd jobs into a file then you'll have them for later. You could of course save the state of the running Lisp, but a source file is better in that it keeps a nice tidy record of the source code of your hacks.
Serious programs written in Lisp are mostly organized pretty much like serious programs written in other languages: the program is factored into a collection of source files that, in the best case, reflects a logical decomposition of the functionality. There's some form of system loader that compiles and loads the sources in the right order and, if desired, dumps the result into an executable program.
Nowadays most people use ASDF for system loading and dumping, I think, but it's not hard to write your own system loader, and I still know people who prefer to do it that way. Just as an example, CCL still uses its own homegrown loader to build.
If your program is like a Python script then it's a Lisp source file that you pass to the Lisp kernel, in just the same way you would do it in Python.
If it's a compiled executable, then your sources were compiled into the Lisp and dumped to that executable, and it works pretty much like a C program. The main difference is that there is no distinguished main(); instead, Lisps generally treat the Lisp's own repl as the default main function, and the image-dumping tools offer you the option to substitute the main function of your choice.
Things are a little different, but only a little, and it's pretty easy to learn how they work.
To recap: when you load a (dynamically-linked) C program in Linux or OS X, the OS will create a new process, and map the dynamic linker (ld-linux.so or dyld) into that process's memory. It then transfers control to the dynamic linker. The dynamic linker then loads the rest of the program into memory (from ".so" or ".dylib" or ".dll" files) and links everything together (i.e. resolves symbol addresses) before passing control to the application entry point.
A compiled Lisp system like SBCL works much the same way, except it uses its own linking/compilation machinery. A new process is bootstrapped by running a C program, which loads the Lisp image into memory. The image is compiled code/data containing the Lisp runtime, compiler, and standard library. After the Lisp image is loaded, you've got two things to work with: COMPILE and LOAD. COMPILE takes Lisp source code and compiles it into a FASL (compiled code and data, equivalent to a ".o" file). LOAD functions much like dyld or ld-linux, and loads compiled code and data into the running process. Indeed, in some implementations like ECL, LOAD is just built on dlopen(). The "eventual program" is ultimately built by LOAD-ing the FASL files comprising the program.
In practice, you don't do this manually. Instead, you use something like ASDF: https://common-lisp.net/project/asdf/asdf.html#Defining-syst.... ASDF functions much like 'make' in that you've got a system definition listing the files belonging to your program. Whenever you call ASDF from the REPL, it'll look at what source files need to be (re)-built, compile them to FASLs, and (re)-load them into the running process. In contrast with C (but similarly to Python), Lisp allows code to be loaded into a running image. In the Lisp workflow, you don't restart the program you're writing each time you recompile (although you can, if you want). Instead, you keep it running, and load or re-load code into it as you work on it.
> I'm used to languages like Python, that have a number of files that are modules, and to start a program you run one of them as an entry point.
You can do that. start a Lisp with a file to load at start, it can then load the dependencies.
> C programs consist of a lot of files that are compiled and linked into a binary executable.
You can do that, too. That's typical when you create a 'system' declaration which describes your software. ASDF would be a tool for that. You then compile the software, load it and dump an image. Some implementations have a more elaborate way to create applications or can create loadable libraries, which you can integrate into other applications.
> You build an in-memory state by adding things to it, later dump it to a binary. How do you get an overview of what there is?
Typically you would work with files. Write the code in files, evaluate the code from there and build the software from time to time as a whole. If you use SBCL, then you get tons of compile time information, type checks, efficiency hints, warnings, ....
The difference with Common Lisp is, you are not limited to this build model, but while your whole code is loaded, you can keep redefining functions for development. This creates the minimal cycle of editing and testing any single function in your program. This is a big asset in development, but in no way means you have to build your systems that way. Actually the contrary, I can only recommend to restart your lisp image frequently to ensure the system loads and builds from file rather than depending on the image state.
Code goes in .lisp files, and then you either load it into the REPL or pass it to the compiler and tell it what the entry point is.
I suppose it's technically possible to develop an entire application at the REPL and then dump to a binary, but I don't think anybody does that any more than they do it in Python.
You can still look at the files that get loaded. Lisp organizes itself around systems (libraries) and packages (namespaces). It's good to check what packages a system has, then you can check out the symbols provided by a package.
Lisp isn't totally wild-west in the concept of the Lisp image. There is organization to good code.
C programs consist of a lot of files that are compiled and linked into a binary executable.
Whenever I've tried to learn CL, I couldn't really wrap my head around what the eventual program would be. You build an in-memory state by adding things to it, later dump it to a binary. How do you get an overview of what there is?
I'm just too used to my files, perhaps. Or I'm missing something.