It does depend a lot on your background, because there are large sections of the book that rely on examples taken from electrical engineering and mathematics. These could be tough going if you aren't that familiar with calculus, complex arithmetic, or simple digital circuits. I think it's the biggest flaw in the book, and others have criticized this as well.
That said, I really hope not to scare anyone off! You can skip things you struggle with and come back to them, but I do think at least reading the whole thing and attempting all of the exercises is really valuable (there are over 350 exercises, of varying difficulty).
So I would try it out, read through the first chapter and see what you think. You may not find it as magical and wonderful as others have, but you might. And you have to ask yourself why you're reading it. You might find that you don't want to bang through it as fast as you can, just so you can tell your colleagues you read it. You might find that it opens up a whole world of alternative programming languages, interpreters and compilers, and it might just be the opening of the rabbit hole, that you won't find the bottom of for years. If that's the case, the great Lisp books: PAIP, LiSP, EOPL, AMOP ... they're all waiting for you. At least, that's what's happened to me.
In the the MIT class, you were expected to put in 15 hours a week for 14 weeks, which is the length of an MIT semester. I probably put in double that, because I loved the materials and exercises so much. (Due to my efforts I did get one of the very best grades in the class, out of 200+ students, but it wasn't necessary to work quite that hard to do well in the class. I'm sure that one could have learned a great deal with the recommendation of 15 hours.)
I'm sure that you can still get a great deal out of the book just by reading it carefully, and that would take a small fraction of the time that it would take to treat the book like a real class. But the best way to learn, of course, is always by doing!
P.S. Be sure to watch the lectures for the class, which are online, albeit in a somewhat abbreviated form. In fact, it might be best just to start with them, and see how you like them.
Also, yeah, read the entire thing! You could skip some sections, but I would advice you to mostly move linearly from the start.
Maybe a month or two, depending on how much free time you have and how many exercises you decide to do. More if you're an exercise completionist. I only did the ones I thought sounded interesting, honestly.
How long would it take someone of average programming skill (with no lisp/scheme background) to bang through this at say 4 hours a week? Is it even worth reading the entire thing as opposed to a few chapters?