There are lots of companies offering software to do exactly what you describe, that is, statistical analysis of telemetry as a way to detect anomalous behaviour e.g. LogRhythm