When Finder is active click on the View menu, press the option key and select "Customize Control Strip", then you can just mouse down to the touchbar and remove the Siri button.