Completely deactivate it

If you choose to disable Google Assistant completely, follow these steps:

    Open the Google app and select Menu (the three bars)
    go to Settings
    Google Assistant section click Settings > Phone and disable the Assistant option

I have a custom domain and disabled it for my family:

Switch off Google Assistant (for entire custom domain) Click Search and Assistant. Tip: Scroll to see all services. ... Control who uses Search and Assistant in your organization

    At the left, select the organizational unit.
    Select On or Off.
    To keep the service turned on or off even when the service is turned on or off for the parent organizational unit, click Override.