A virus needs only a handful of genes/proteins to work, one of them is a "virulence" factor that binds to a receptor on the cell which starts the cell entry and infection process.

CoVID-19 is a much more dangerous disease than other coronaviruses because it has a spike protein that binds to the ACE-2 receptor on your kidney, heart, blood vessel, and yes lung cells. Other coronaviruses have different spike proteins that bind to different receptors and don't seem to kill people effectively as the CoVID-19 spike protein.

CoVID neutralizing antibodies target the unique spike protein so the fact that other coronaviruses don't present a durable target isn't relevant.