We spent a ton of time thinking about this. We have an "op executioner" in our tensor library that handles special cases like this. We call it "grid execution" where we look for opportunities for grouping ops automatically. We will be combining that with our new computation graph to automatically look for optimization opportunities like that.

Right now we hand write all of our own gradients as well.

The overhead can come from a ton of different places. This is why we wrote workspaces: http://deeplearning4j.org/workspaces

Allocation reduction and op grouping are only a few things you can do.