There is a book, Applied Mathematics for Database professionals that uses a terse mathematical notation to describe queries and I remember actually kind of liking it. I especially like that they used it to describe the state of the database and the transforms.