Most notably if it was just gravity then we'd expect the effect to be more homogeneous. But with dark matter distribution it isn't. The distribution of dark matter is highly non-homogeneous and acts just like matter does, creating webs and clumps. It would be weird for a field to do this, especially since fields are associated with particles (we still haven't found the graviton, which is the exception to that statement, so far. But we also don't expect to find it without a substantially larger accelerator. One that would be difficult to build on earth)