If you know that you want to hit points A, B, C, ..., in your talk, and you know your material cold, then you don't have to memorize how you want to connect A and B, B and C, etc. That is what you re-create on the spot. The practice is important because sometimes you can't connect two points on-the-fly. Practicing your talk at least once will reveal those places, and you can either: figure out what the connecting bits are, or decide you don't want to cover that.
In addition to the jokes and wry remarks, I find that I also tend to replay the same body language in talks that I did in practice.
FWIW, when I ran for office, I never memorized a single speech, nor did I use notes. I mentally rehearsed the key points and structure on the way to the venue, and then improvised. Take the fact that I didn't get elected however you want, vis-a-vis the effectiveness of my approach. :-)
It's confidence in your ability to convey your point. It's confidence that your knowledge of the topic is solid. It's confidence that your audience is capable of understanding you. It's confidence that you've put the time and effort into making sure that all the previous stuff is true.
Instead, you want to memorize all the key features, jokes, and wry remarks.
You can write out your speech, if it helps. But read it. Out loud. To a wall. About ten or fifteen times. Read it, without looking at the paper (except when you forget what goes next), and without trying to duplicate it verbatim. You'll learn what phrasings come naturally to your tongue rather than your hand, and it'll flow a lot better when you give the speech itself.