The first layer, yes, but after that come non-specialist academics, who aren't qualified to judge my particular work on its merits, and so have to substitute some other metric; and then administrators, whose fitness to judge the merits of any academic work is, let me say, not necessarily a given. For example, even without any worries of bias: as a mathematician, it is always a struggle to get non-mathematicians, even (or perhaps especially) other scientists, to realise that author order on mathematicians' papers reflects only one's place in the alphabet, not one's contribution.
It really is bewildering.
Online academic communities and publishers such as ResearchGate and EndNote could come up with potential solutions here. ResearchGate already asks us to effectively vote about other users' authorship and expertise. Adding a voting system (or similar) whereby users declare the roles played by each author might help dismantle author position hegemony.
Voting might be imperfect, and perhaps there are other options. The current system, though, is ridiculous and contributes towards the exploitation of junior academics, and poor inter-discipline communication. Clearer computational representation of roles could also improve search.