I don't know whether or not it has always been the case, but Signal works fine without "access to my contact list". The Android app does seem pretty persistent in asking for it, though!