I've wondered what the ideal time of effort is on several occasions. Sometimes I'll get stuck on an algorithm problem and no matter what technique I try I can't make progress. Sometimes I'll try for days, until I eventually just give up. That's obviously not ideal, but I think 5 minutes is too short for this situation. I worry that I won't internalize the solution or technique very well unless I've wrestled around in the problem space for a while first.
The best way to combat this is to be smart about using whatever resources available to get an expert to help you. Post questions on sites like mathoverflow/stackoverflow, and be persistent. You could waste time spending weeks on a single problem that someone with experience can solve in seconds.
Personally, this is why I don't like self-study on it's own. If you can form a study group, do a part-time degree or find some other method to ensure that you don't go down costly rabbit-holes, you should. The gains in efficiency will be worth it.
This is your secret.
* Acknowledge that sometimes the book you're reading is repetitive or not very well written. If you're <10% in, read up on reviews and try to find a better book on the same subject. If ≥10%, commit and complete, acknowledging that you're not necessarily the best judge of a book's quality.
1) Every weekday 30min uninterrupted study at a specific time, preferably in the morning. Don't miss, rain or shine. Take weekends off, though.
2) Work in a strict linear fashion: read a chapter, then solve all of that chapter's exercises in order without skipping any, and only then allow yourself to peek at the next chapter. Don't "take a first pass through the book", none of that. To remember where you are, use a bookmark.
3) If you're having trouble with some exercise, you can look up the answer key for that specific exercise, but only after you've spent 5 minutes of effort on that specific exercise without making progress. After that, the same rule applies to the next exercise.
4) You've got to get to the end of the book. Matter of pride.
For me this approach has worked well for established topics in math, physics, econ. Haven't tried it for CS or ML.