You are still exposed to Google/Apple, since it can easily combine your contact information with traffic information and with great deal of probability detect who was the recipient.
I still would take it over encrypted SMS, where this data is open to carriers and carriers are basically perma-owned by everyone.
1) Google can see Alice send message via Signal
2) Google can see Bob receive message via Signal
3) Google knows that Alice is in Bob's contacts, and Bob is in Alice's contacts
Those three pieces of data should be enough. I don't see why Google can't do time correlation of those messages. Both Alice's and Bob's contact lists are very finite. It might be not obvious for single message, but for series of messages (and people chat this way) it should be very obvious who is talking to whom.