It's discussed extensively in History of the Lord of the Rings. The sequel was originally intended to have a tone as light as The Hobbit (which somewhat excuses his original "Bingo" Baggins). Tolkien had no idea where he was going when he started writing; he just knew he wanted to start with a party and somewhere include an off-hand story involving Tom Bombadil. He eventually decides that the protagonist (Bilbo/Frodo/Bingo/whoever) will go to Rivendell.
He wrote a few chapters with all this in mind, still in a light tone. He went back to rewrite some things and he decides to add a passage where Frodo hides to surprise/scare Gandalf, who is coming down the road on his horse. Tolkien writes the part where Gandalf stops and starts sniffing, probably intending for Gandalf to ruin the surprise.
But then, with no known explanation, he scratches out Gandalf and writes "black rider" (or something close to that). In the margin he writes "who is the black rider?". He then pauses his rewrite so he can ponder who this new character is. This starts him down the new, darker, path that leads to the Ring and Sauron. The sniffing rider eventually becomes a Nazgul. Fascinating stuff!
His inclusion in the book is awkward and people have tons of questions about it because it doesn't really fit. Tom Bombadil is more of a nursery-rhyme character, and would have fit better into a Hobbit sequel than what became The Lord of the Rings.
His continued inclusion reflects the way Tolkien was loath to give up an idea once he'd written it, leading to a chapter that is at once incredibly evocative and maddeningly uninformative.
We start with the cheerful birthday party in the Shire, and Frodo’s departure — written in a similar tone to The Hobbit as Frodo considers himself to be retracing Bilbo’s steps and beginning an adventure of his own. It starts to get darker and more dangerous (but still with Tom Bombadil around to save the day, at least at first); and once the hobbits leave the Shire they suddenly no longer feel safe or in control. By the time we get to Rivendell it’s clear that the hobbits are way out of their league, and that the plot is way bigger than just Frodo and his friends. And as the story develops, and the hobbits split up and become involved in the war from different areas of Middle-earth, this change in perspective continues and the tone continually gets grander and darker. It’s like a “zoom-out” effect that starts at the birthday party as a carefree fairy-tale and culminates at the destruction of the Ring as a massive, high-stakes epic fantasy.
It’s interesting (and totally makes sense to me) that the story developed this way in Tolkien’s mind too; and I think it’s clever that he decided to make it a part of the story rather than rewriting everything to have a “consistent” tone. I think it makes it a lot easier to follow the story and a lot easier to get to know the characters than if it had started like RoTK from page one.
They left as comfortable town people, scared of the world and in need of help; they return as veteran adventurers capable of dispatching the local evil on their own.
You can read the early drafts in book 6 of the History of Middle-earth, and watch as "the tale grew in the telling". They start with Bilbo, then the protagonist changes name and identity as the concept fills out. He made it as far as Bree in a very Hobbit-esque style before scrapping it all and rewriting it in the more adult LotR style.
(He later tried to re-write The Hobbit in that LotR style, but it was dreadfully boring and he gave up after a few chapters.)
I wonder what the publisher's thoughts were about Tolkien's response: 'How about a six book, three volume epic, not suitable for much of the same audience (children), in essentially a new genre, with almost all new characters?'
For example there was a 17-year gap between The Hobbit and LOTR's first two books.
Tolkien pulled much from older myths, of which Tolkien was a leading scholar. The Ring as a driving force (and its power and the consequences of that power) was not an invention of Tolkien's; Tolkien didn't get the idea from the Hobbit. Wagner famously used it in the 19th century, and likely myths, on which Wagner also relied, preceded Wagner.
I think your description is a common process of many (most? almost all?) artists and creative workers. Creation is not making things in a first draft out of whole cloth. You don't know where you are going or what will work. It's not unlike a startup in that respect (please let's not get carried away with our frame of reference): you have some great ideas and talent; you develop it, often in unanticipate directions; and eventually you find an application for them - sometimes completely unanticipated, even from a side project.
> He began his book as a direct sequel to The Hobbit
Where does Tolkien describe that?