Good job. It's a decent undertaking for a single person.

From a cursory glance, you might also be able to speed up the CPU backend by getting rid of the shared_ptrs and defining a stricter memory ownership model.