What are you talking about? You do not need phone number. I self host matrix instance and all I did was create username and passwords for people I know and shared with them. They then logged in and changed their passwords. At no point the sign up workflow asked for phone number.

Also, the UI has improved a lot recently. There are number of matrix clients that have appeared on app stores but I still prefer Element which gets regular updates and features.