I've never spent much time there, but some of the ideas are laid out well in this two part post: http://blog.ksplice.com/2010/04/libc-free-world-2/
Doing it with _just_ a hex editor (ie writing out the ELF headers and sections yourself) may not be exactly what the commenter did, though. Cos that sounds really tedious.
I imagine that even writing "Hello, World" in the simpler way that you describe, on a simpler system (Apple IIe or so), would probably teach me something. I remember trying to write 6502 machine code in hex (no assembler) on my IIe when I was about 10 years old. It never worked then, I wonder if it would work now?
http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...
The idea is that you strip away everything that does something for you so that you are sure to completely understand the entire process with no "pushing the green compile button."
Does this involve anything more than loading "Hello, world" into memory, setting up the registers/stack in a certain way, and invoking the syscall interrupt? (All the cool stuff happens in your terminal emulator. That's where 0x41 gets turned into the pixels that look like "A" on your screen, after all.)