I just set it to a speed on the highway... and the speed stays there until I brake.
Are you talking about adaptive cruise control (ACC) to follow the speed of the car in front?
Funny, I don't feel the same -- my foot rests below the brake (on the floor) instead of on the gas pedal so the time to move it to the brake feels roughly the same, and I'm paying 100% attention while driving since I'm still steering and constantly monitoring distance to the cars in front of me. So I don't notice any less attention -- it's just a rest for the muscles in my right foot.
But really I'd definitely go with your own perception here -- if you feel like you're paying less attention then it's a good thing you're not using it! Everyone's attention habits and patterns are different. And I'm glad to know people vary in this.
I had the power steering drop out going into a turn (fuel pump failure IIRC), and it could have easily caused an accident (I physically couldn't turn the wheel enough to complete the turn).
Or you have very serious muscle weakness, at which point I'm not sure if driving a car is a smart option in any case.
Cruise control frequently fails and lane assist that explicitly the one that take steering control away from you has put me in dangerous situations more often than it has saved me from them.
Lane assist as an alarm to alert you is great though.