That said concerns about the sensitivity of information are absolutely valid and should be at the forefront of EMR systems.
In my experience (UK) access to records is reasonably well restricted to relevant staff. Access is audited in efforts to identify unauthorised access and this is taken extremely seriously; for example opening records for patients not under your direct care with no valid reason.
Further, for particularly sensitive information such as some psychiatric histories, or medical photography of sensitive areas, this is kept behind a secure area within the EMR systems, not generally accessible. Opening it requires signing a declaration that you have either discussed it with the patient and gained consent, or that you do not have consent and you are opening it in the patients best interests in an emergent situation as they are unable to consent (e.g. history of depression with paracetamol (acetaminophen) overdose brought in unconscious by ambulance.
EDIT: left original link tried to clarify.
[0]: "I drive 300 miles to seek care and always pay in cash." http://www.idealmedicalcare.org/75-med-students-antidepressa...
It is also completely off-topic.
See: https://journals.tdl.org/jrwg/index.php/jrwg/article/view/97
https://www.dailydot.com/irl/trans-broken-arm-syndrome-healt...
https://www.pinknews.co.uk/2015/07/09/feature-the-dangers-of...
https://www.reuters.com/article/us-britain-lgbt-health/briti...
Sure this is a minority, but it still harms people.
More pressingly - how can a doctor provide the best care in the absence of records?
Imagine trying to ship a bugfix for a complex system where you don't have the source code and the business owner is refusing to give you the documentation but will sue you and ruin your career if you make a mistake.
I'd probably refuse to see the patient given how much pressure doctors are under to not make a mistake.
What happens if you got a nasty gash and wanted the wound treated and the doctor wanted to issue antibiotics as a precaution but you had a penicillin allergy (which you either forgot to disclose or simply didn't know about but were tested for at some stage in your childhood)
you shouldn't be. to a certain group of people, it is particularly loathsome to be prohibited from doing things that can only hurt the person doing them. this is basically how we treat children.
a patient should be allowed to limit access to their medical records for pretty much any reason. an individual doctor should also be allowed to refuse to treat a patient who is unwilling to disclose part or all of their medical records; they may have valid liability concerns. but revealing your records should not be a precondition to accessing the entire field of medicine. what part of this do you have a problem with? you still get to deal with your doctors however you want.