The problem really is that you can't divide by zero, even in an algebraic expression.
A simpler example of this phenomenon (which blew my mind when I first encountered it) occurs with the equation x = x^2. If you divide by x, you get x = 1, which is a solution to the equation, but where did the other solution x = 0 go??
Whenever you divide an equation by an algebraic expression, you need to consider the possibility of that expression being zero and treat it as a special case. So in the case of x = x^2, you can reason as follows: maybe x = 0, in which case … what … ah yes, that's a solution! Or maybe x ≠ 0, in which case we can divide by it and get x = 1. That doesn't contradict the assumption x ≠ 0, so it's okay, and x = 1 is the other solution.
So, the actual problem is dividing by zero. Your assumption that "we are dealing with algebra and not numerical values" is false because it completely ignores the "if x and y = 1" part.
You can't divide by 0 you just get nonsense.