This has always been my issue with it. For some reason it doesn't have pressure sensitivity like the trackpad (which also has no physical button). Lightly brushing the touchbar is enough to activate the controls on it.