Recent research has shown that progressing with 8-12 reps is not any more optimal for hypertrophy than progressing with 20-30 reps (what people would normally think of as developing "muscular endurance"). The difference in practicing 8-12 reps is increasing strength as well, i.e., participants doing 20-30 reps didn't see significant increases in their 1RMs (despite muscle growth) while those doing 8-12 reps did.
Stopping one rep short of exhaustion is an important technique: It doesn’t decrease your rate of gains and it increases your recovery time between sets so that you can complete all your reps (study, study, study) [3, 4].
#2: Shrinking from overworking is surprisingly a real thing. The science is unclear as to how exactly muscle mass responds to resistance exercise, so my answer would be made up if I gave you one. What I can tell you is that "being overworked" is the result of lifting too heavy for too many total reps. This might seem impossible since your body should be unable to continue lifting past a natural point of exhaustion, but if you use machine exercises (especially a pulley machine), it's easy to cause a disproportionate amount of stress to your muscle relative to the incrementing heaviness trajectory you were lifting for free weights.
[1] https://www.researchgate.net/publication/7873571_Muscular_ad...
[2] https://www.researchgate.net/publication/23252049_Changes_in...
[3] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4731492/
[4] http://www.ncbi.nlm.nih.gov/pubmed/21986694