I suppose some sort of icon (like an arrowhead) next to each link could help them to afford clicking.

I doubt it, blue and underlined is pretty much a standard. I think it looks perfectly fine and clear. There's a reason most browsers' default stylesheet will make links look like that.

Underlining links (particularly where there are many of them) can harm readability, which is why many popular websites do not do it, including the major search engines, Wikipedia and even Hacker News.

Not having a followed link style in this context degrades the usability of the website because you have to rely on your memory of which links you have followed on the list and which you have yet to see. This is an important and little remembered point in the current interaction design "dark age", as Bruce Tognazzini puts it, and Norman et. al. <a href="https://www.fastcodesign.com/3053406/how-apple-is-giving-des... recently written about</a>.

(It is of course also true that the unfollowed link style is not very clear)

But I absolutely agree with you that visited links looking like unfollowed links is a usability issue.

---

Edit: Added an example.

Of course, you could call the heading 'Links', but that shouldn't be necessary -- the links themselves should afford clicking. It would be like putting a label called "push" above a door handle when the handle itself should afford pushing.