You are a collection of chemicals wandering around by stocastically seeking the highest entropy state you can achieve, so yes, you are a sort of Markov chain I suppose.

But so is a rock, so I would argue it is not an overly illuminating fact. Furthermore I suppose you have this very convincing experience of free will, an experience that seems pretty robust even when facing such facts as the brain is just made of chemicals, so I wouldn't worry about it.