a) Permutation symmetry of identical atoms b) Rotational symmetry of system c) Translational symmetry of system
I suppose it could learn them approximately, given enough examples, but why bother? I think it'd be kind of like not using a convolutional neural net for recognising digits in photos and just using a bazillion more weights and examples.
I'd say you'd start with a completely general learning model which respects the above symmetries and then see where it takes it.
However, I don't know how you'd make a transferable N-body potential from a model taught only on some number of atoms. Again, I guess it's kind of like training a CNN handwriting recogniser on 256x256 images and then applying it to arbitrary sized images, which you can only do by assuming locality and translational symmetry of the features.
You say that: "Without the above simplifications like modelling it as a sum of local potentials, and making env(R) a cutoff, you would indeed just be fitting a 3N dimensional function in the case of N atoms. It'd be exact, but it'd also blow up pretty badly and utterly nontransferable. Also, the energy surface isn't necessarily continuous and differentiable -- consider the energy when two atoms move to occupy the same position."
Those contraints and representations of the problem are the thing I would want the machine learning algorithm to discover. Is this beyond the scope of the state of the art algorithms in machine learning? I understand that making a good choice for the representation of the problem should make the job of the learning algorithm easier, but finding the representation that is best is quite challenging.