If this were a restaurant, that would be a different story.
> Exceptions (one-party consent required): (1) where there is no expectation of privacy, (2) recording within government proceedings that are open to the public, (3) recording certain crimes or communications regarding such crimes (for the purpose of obtaining evidence), (4) a victim of domestic violence recording a communication made to him/her by the perpetrator (for the purpose of obtaining a restraining order or evidence that the perpetrator violated an existing restraining order), and (5) a peace officer recording a communication within a location in response to an emergency hostage situation.
Source: https://www.mwl-law.com/wp-content/uploads/2018/02/RECORDING...
No legally protected right. This doesn't mean it is ethical, and given that it is a protected right in other jurisdictions shows it deserves more consideration and should not be hand waived away.
If "it's legal" is the argument being used to defense a behavior, it's safe to assume it's not actually a good one.
While I think this is a really cool project, I also agree with the privacy issues. CA is a two party consent state, and recording a conversation (which this is likely to do) like this is likely illegal. While a person might not have a expectation of privacy about someone just hearing the conversation, they are protected by law if they are recorded without their knowledge.
NB: I am not a lawyer, and the above could very well be wrong.
Edit: As I was informed below, I was wrong on the legal points.