There should not be linearly increasing memory usage. If you find memory leaks like this, please file bugs on github issues. AFAIK there currently aren't any.

Of course, if you do something like keep an undo stack around (check out swannodette's blog for a great example of this) then of course you have linearly increasing memory usage. One great property of persistent data structures is that the memory usage will be much smaller than if you kept your undo stack as copied JS values because of structural sharing (see one of swannodette's recent talks for a great explanation of this).

In practice, if you're implementing an undo stack, you should remain conscious of memory usage. You may want to only keep a fixed maximum number of previous states around so you don't have linearly increasing memory use.

Do you know of a way to limit your undo stack not in terms of undos (snapshots) but in terms of memory usage? i.e. limit the undo stack to N megabytes of memory?

Furthermore, do you know of any ways to perform automated testing for memory leaks?

In production, no. In a development environment yes. AFAIK, there is no exposed API in the browser to describe the size of the heap. However, you can expose hooks to get this information in a v8 or node instance; snap heap size, run tests, run GC, snap heap. Compare before and after in your test runner.