If you want command line, then you can edit the /etc/rc* files to startup into a full-screen terminal if you like. There's nothing that says you must run Gnome/KDE/etc.

Karmic Koala Beta runs on virtually everything I've tried it on with no driver problems, even wierd built-in webcams on notebooks.

If you want to go really minimal, then you could run Debian (upon which Ubuntu is based). I have a Compaq Armada running Debian without problems.