> It annoys me that apostrophe and English closing single quotation mark are the same character in Unicode.

They're not. You should be more than just tempted to use ' (U+0027 APOSTROPHE) for apostrophe, because it is. Closing single quotation mark is ’ U+2019 RIGHT SINGLE QUOTATION MARK.

> When text is set, U+2019 right single quotation mark is preferred as apostrophe, but only U+0027 is present on most keyboards. Software commonly offers a facility for automatically converting the U+0027 apostrophe to a contextually selected curly quotation glyph.

https://www.unicode.org/versions/Unicode14.0.0/ch06.pdf

Typographically, U+2019 is the appropriate glyph for quotation marks in English and in some other western languages.

That doesn't reflect actual practice any more than the previous Unicode recommendation to use U+02BC MODIFIER LETTER APOSTROPHE (which at least had some semantic justification in that quote marks are not part of words). Yes, typographically an apostrophe looks much like a right single quote (although the position may be different); in the current century fonts handle that.

> in the current century fonts handle that

Except that they can’t, because (due to the limitations of ASCII), U+0027 had an overloaded meaning, seeing that the character had to cover both opening and closing quotes (as well as being used as a prime tick). Which is why pretty much every font I’ve seen has a straight glyph at U+0027, whereas the glyph at U+2019 is the one you’d usually want for an apostrophe. You can also observe that word processors follow the Unicode standard in this regard. E.g., after typing “they're” into Libre Office, once you copy-paste it, you’ll see that the ASCII single quote has turned into U+2019.

There are dedicated opening/closing quotation mark characters too, U+2018 and U+2019: ‘’