Downloading and executing code is only one way a browser session can be abused. At the very least you're giving away everything your browser (even Lynx) puts in the headers of a request. That's often a heck of a lot of useful information for an attacker. Lynx supports cookies too so it would be possible to track a user between sessions. I don't know how that might benefit an attacker but I'm not an attacker[1].

I think a reasonably paranoid approach like "Hackers might think of ways to abuse this that I haven't thought of" is best. Unless your job is to take a risk and visit a phishing site, don't take the risk. Even with Lynx.

[1] Exactly what an attacker would say!