There is this paper by Joulin & Mikolov: Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (http://arxiv.org/abs/1503.01007).

In this case, the memory of the RNN is an ensemble of differentiable stacks.