Its NN black box alright, for a hand crafted model you would have something like:

    if prediction != reality
     {mismatch+= whatever the time tick is}

    if mismatch > second (or two)
     {abort driver assist}