Essential complexity is inherent to problem being solved and nothing can remove it.
Accidental complexity is introduced by programmers as they build solutions to the problem.
Lisp is nice because eliminates a lot of the accidental complexity through minimal syntax and lists as a near-universal data structure.
As a programming language, Common Lisp is large enough and multi-paradigm enough to allow for elegant solutions to problems. It does require some experience with the language and some wisdom and discipline to know what pieces to select and how to best use them.
However, like all large, multi-paradigm programming languages that have been around for a while (I'm looking at you C++), programmers tend to carve out their own subsets of the language which are not always as well understood by those who come after them, particularly as the language continues to evolve and grow.
There is also the problem where programmers try to be too clever and push the language to its limits or use too many language features when a simpler solution would do. All too often we are the creators of our own problems by over-thinking, over-designing, or misusing the tools at hand.
if the language is not powerful enough to allow for that people will inevitably add preprocessors, code generators, etc... to do the things they want.
This is definitely true and it adds to the accidental complexity of the system, usually to save programmer time or implement layers of abstraction for convenience.
Common Lisp and C++ have both incorporated preprocessors and code generators through Common Lisp macros and C++ template metaprogramming and C++ preprocessor / macros. These features give the programmer metaprogramming powers, enable domain-specific language creation, implement sophisticated generics, etc.
They are powerful language facilities that need to be used wisely and judiciously or they can add exponential complexity and make the system much more difficult to understand, troubleshoot, and maintain.
Some languages attempt to discourage bad taste by limiting the power given to users. Common Lisp embraces the expression of good taste by giving users considerable power.
We know somewhat how to control accidental complexity in systems that are highly constrained and don't let you deal well with essential complexity. And we know somewhat how to give you powerful tools to deal with essential complexity.
We haven't figured out in general how to make systems powerful enough to deal with the range of essential complexity, but constrained enough that this sort of accidental complexity isn't common.
Of course a lot of real world systems don't do a great job of either.
As a result, the simplest system is obtained, but the design process is a systematic project.It is difficult to design a complex system into a simple and smooth pipeline system.
I think "incidental" (non-essential, secondary, happenstance), and "accidental" (non-intentional, happenstance) are more or less synonyms here, not contrasts. I think there, uh, is basically no significant difference between "incidental complexity" and "accidental complexity".
[...] in the real world simple is _always_ a lie [...]