Not true. Windows CE was a solid offering.
Every implementation of this I've seen has been a slow burn disaster. It's still deployed, and it still sucks.
I knew a few people that loved them, but they were not ordinary people.
I'm not saying it was totally useless, or that people didn't find it "fine" for some tasks, but it was like a crippled notebook that could barely do any of those things. The only advantage it had was size, and even then it still weighed vastly more than any of the phones from that era. They still cost a small fortune, maybe half the cost of a decent laptop, but since they weren't a laptop or a phone you still had to get one of those anyway.
Compare with two competitors from that time, Danger Hiptop and Sharp Zaurus, took a very different approach: Keyboard centric, good battery life, limited functionality but focusing on specific things rather than trying to do everything.
As I recall, the pen was fiddly but not necessary; the screen was pressure-sensitive, so you could use a fingertip just fine, or the eraser end of a pencil. Modern capacitative touchscreens are much more accurate, but I sometimes miss being able to use any object as a stylus. :)