Relatively large-scale production required I/O devices to share a common character set. These tended to be based on pre-computer character sets (teletypes, punched cards) and were limited in size both by the I/O devices and by storage requirements (don't forget how small and expensive memories were). A lot of work went into choosing character sets.²³⁴
1963 ASCII had an up arrow ‘↑’ dropped in 1967 in favour of ^ (because furriners wanted accents), and a left arrow ‘←’ dropped in favour of ‘_’.⁵⁶
¹Jean Sammet, Programming Languages: History and Fundamentals 978-0137299881 ²Charles E. MacKenzie, Coded Character Sets: History and Development 978-0201144604 ³http://www.trailing-edge.com/~bobbemer/SURVEY.HTM ⁴R W Bemer, Design of an improved transmission/data processing code http://dx.doi.org/10.1145/366532.366538 ⁵American Standard Code for Information Interchange http://www.wps.com/projects/codes/X3.4-1963/index.html ⁶Revised U.S.A. Standard Code for Information Interchange http://www.wps.com/J/codes/Revised-ASCII/index.html
Languages which support this (including Agda, I believe) usually support some fallback to ASCII, allowing you to type "->" instead of a Unicode arrow, because it's easier to type on the vast majority of keyboards.
Of course, APL also famously uses all kinds of symbolic nonsense.
For example: https://github.com/ehamberg/vim-cute-python
It can't be because physical keyboards wouldn't have the key, because my keyboard doesn't display a # symbol, but that is widely used.
I can only guess that the character sets and "what characters are easy to type into a computer" became standardised early on, and no individual language has been able to push such a change (Early in development a language has few users, thus no influence; Later in development, the language has been using "->" itself for a while, so it's less easy to change).
Also, proper languages like lisp use parentheses (), not curly braces. :p