Yeah the reality is that the size of the cake changes, and slicing the cake in different ways can feed back to the growth of the cake.
The perceived value of each slice relies a lot in the psychology of the public.
It's the job of the central bank to make sure the slices are proportioned properly and the perceived value of each slice of cake is maintained.
There are other reasons that inflation is good too, having to do with sticky wages and recessions. I'm not sure how to work that into the cake analogy though.