Should you have the right to watch which direction I'm going when I pass by at an intersection?
How about at the next intersection?
And the one after that, and the one after that, and the one by my home, and all the ones that I happen to go by when I next leave and go somewhere?
There's no single point at which passive observation turns into stalking but we still have laws against stalking and it's still perfectly ok and legal for you to watch where I'm going. If you understand why it's ok to look around you (and perhaps even take notes or draw what you see, snap a photo) but not OK to do that systematically around someone, you should also understand why we might want to restrict automated unwarranted and consentless data collection, even if taking some notes is OK.
The other thing is scale. Laws against seemingly minor things are enacted when that thing becomes widespread enough to upset many people. You probably don't upset people too much by taking some notes in a shop. If every shop had a fleet of staff dedicated to the same thing, that probably would upset people and lead us to a similar discussion.
In particular with data, the processing method makes a qualitative difference, between individual disconnected people each knowing one small snippet of your life, and a megacorp having an overview over whatever you're doing at every moment. By de facto limiting it to "one brain", the panopticon is prevented.
Your rights end where they start impacting other people's rights, and drawing those lines is what we have governments for. Even if you were to use just your memory, if you were tracking what someone is doing in such detail as ad companies do, you'd likely a) not be able to do that to more than a few people b) receive restraining orders for stalking.
Augmented capability lets you cause more, newer and larger problems.
We didn't have speed limits on the roads until humans acquired the technological capability to go faster, causing more fatalities.
If you keep it in your head, it's not going to be stolen or abused. If you put it in a notebook, the risk increases, but you at least aren't going to be doing this at a large scale, simply due to physical limitations.
Various laws have different thresholds, but usually it's something like a "systematic collection of data" or "automated processing" (I think it's the latter for GDPR), which seems like a reasonable compromise to avoid hampering the low-risk small scale use cases.
Edit, looked it up, this is the definition GDPR uses:
This Regulation applies to the processing of personal data wholly or partly by automated means and to the processing other than by automated means of personal data which form part of a filing system or are intended to form part of a filing system.
So if you keep your database in a notebook, you're probably still fine, because the structure makes it impossible to do nasty things at scale. Once you switch to alphabetically sorted index cards, you've crossed the threshold.