If you use something like Lisp's syntax, then writing a parser is pretty simple:

http://blog.fogus.me/2008/07/22/broccoli-v022-bellwitch/

Smalltalk has only about 8 terminals and nonterminals in its grammar. You can hand code a top-down parser for it in an afternoon.

Then there's always Forth.

Lexing/parsing is one of the easiest parts of writing a compiler these days. With a parser generator, it shouldn't take you more than a couple days to write the lexer and parser for a moderately-complex language.

I second the comments up-thread about error-reporting being the hardest part of a real compiler.

"how much stuff is dictated simply by convention"

Hm, like what?

just to name a few: argument passing, process scheduling, register allocation, big vs little Endian - I realize this is a weak list but personally I expected more stuff to have some reasons behind the decisions.

Those are not decided by arbitrary conventions. They're usually chosen by some sort of criteria like performance, simplicity for teaching, portability, etc. There's definitely flexibility in choosing the specifics, with a lot of trade-offs to consider, but they do have specific design criteria and are not just conventions. They are conventions in the same sense that TCP/IP is a convention -- since if everyone did something completely different, there would be no interoperability.

Calling conventions (argument passing protocols) are designed to be efficient for a particular CPU architecture. Scheduling algorithms try to make the best use of all resources while still meeting other performance criteria (e.g. firing events at roughly the right time). Register allocators are usually trying to produce the fastest code possible. And there are arguments both for and against big and little endianness, but I don't remember what they are. :-)

I got exactly the same impression from my compilers class, especially for parsing - it's a bunch of techniques, which vary in their applicability. I still have that opinion, after years of research and commercial work using parsing.

Although there's some mathematical theory in the background (regular expressions, context free grammars), the techniques don't fully correspond to them, and aren't really implementations of the theory, but a convenient, useful, doable subset. The adoption of the techniques seems to be dictated by what works to solve specific practical problems. The techniques themselves are pretty ad hoc; but they have been studied, and there is a body of knowledge about where each technique is applicable. Although certain correspondences have been proven, it's not neat and beautiful like physics; but truth and beauty aren't what ye need to know here.

It's really "Compiler Engineering".