HIPAA compliance has a lot of ambiguity. You are correct if, say, your employer requested medical records from your doctor. However, any entity that works with your provider that is facilitating its business can still get your data without your consent.

JohnFen's comment is correct. This is why 42 CFR Part 2 was created; HIPAA was too loose with patient data.