Actually, a whole lot of recombinant approaches have moved to human trials (along with viral vector approaches). They early ones showed no efficacy; the newer ones we just don't know (yet).
The main benefit of mRNA is iteration: you can try a whole lot of different protein mixes and see what works best in an animal model. A secondary benefit is that the sustained churning out of proteins for a few days seems to generate a much broader antibody response, making getting some of those elusive bnAbs more likely.
A big problem with killed HIV vaccines and purified fragments of HIV is that growing virus in human immune cells is extraordinarily costly, and you really wouldn't want to have some live HIV leak through your purification process. So this is why we're mostly talking about other approaches to make proteins (viral vectors, recombinant DNA in bacteria and other cells, mRNA vaccines, etc)
I should have been more clear here, I meant specifically trials of what I presume could have been attempted if the vaccine was expressed as protein instead of mRNA. But thanks for the clarification. And the mention of viral vectors which are an area I was less aware of till recently but it has been cool to see them in more places.
Yes, and one vaccine approach is getting a bacterium or other virus to make a lot of a protein you're interested in, and then purifying the stuff you're interested in and forming it into a vaccine product. These are recombinant approaches. E.g. https://en.wikipedia.org/wiki/AIDSVAX is exactly what you describe: a viral protein vaccine developed using recombinant approaches from e.g. Chinese hamster ovary cells https://pubmed.ncbi.nlm.nih.gov/8142140/
The first was a hepatitis B vaccine developed in the 1970s where yeast cells were modified to make hepatitis B proteins. Then the HPV vaccine used the same approach, choosing proteins that would spontaneously assemble into a virus-like particle that triggers a strong immune response.
Complicated proteins have to come from a living thing, in practice. So if you're going to administer a viral protein, it has to come from purifying virus that you've grown or from a recombinant approach of some kind (and growing HIV is problematic for multiple reasons).
mRNA vaccines are particularly useful because you can substitute bases with N1-methylpseudouridine which makes them identify as "self" by TLRs and doesn't generate counter-productive innate immune responses. You do need an adjuvant so that the cell produces more MHC displaying the antigen and attracts lymphocytes but the nanoparticle works to do that.
The result is a very small message payload to the immune system that targets the correct system. Viral vectors all have more "stuff" all over them which are going to trip other parts of the immune system (or cause side effects like the PF6 interaction with the Adenovirus capsid that is responsible for the rare clotting effects).
However, the mRNA vaccine may be more stable in a way that makes it easier to make or store or for entrance into the body. Not to mention purification steps may be easier. You don't need to make this in some exotic cell line which creates a protein soup you have to purify. It may be easier on the body (less likely to develop a severe immune reaction), or it may be better at generating a sufficient immune reaction because a lot of mRNA development has gone into finding molecules that are readily taken up by the immune system regardless of cargo.
These are a few reasons I can think I am sure I am missing some and there may be a stated answer out there that I have yet to search for.