Sounds like just a fancy way of giving your students unit tests. Our early CS classes will have unit tests but later the students are told to write and submit their own. I think that experience is more intuitive than a black box grader.