But we still have not made clear what the heck the assumptions are.
Confusion matrix: Wikipedia has nothing on it, but Matlab's documentation does.
But there's a problem: Suppose we have some data on cats and dogs and want to have a 'classifier'. So, we do what Matlab said: Take half of the training data and build a classifier. Then we test the classifier with the other half of the data.
But suppose the classifier finds three classifications instead of just the two, dogs and cats we already know exist. So, if we apply such methods where we don't know in advance the number of classifications we might get three classes and be quite wrong and not know it.
Again, the methodology looks like just pushing some data into some software with too little in assumptions up front and too little about what we are getting out on the back side.
- K-fold
- Confusion matrix