At which point you run into problems of defining agency/free will.

Free will lies at the particle level. If atoms are defined by strict physical rules that are deterministic and have no free will then if our brains are made of of the same atoms does that mean our brains are also defined by the same strict physical rules?

What if atoms are not defined by deterministic rules? What if it's defined by probabilistic rules? Then are our brains bounded by the same probabilistic rules?

It seems that given that all brains are made of the same stuff as the universe, the question of free will is more of a physics question. What is the universe made of? And does this fundamental unit allow for free will?

I was alluding more to the issue of deciding how responsible people should be held for their actions, for societal purposes.

That's is one brilliant observation - love it!

there's also subconscious lying, such as hiding our blind spots where the optic nerve is attached to the retina and turning off our vision when we move our eyes, and toying with our perception of time passage to make it seem like there was no gap in vision.