What is a patch in your case? Are you running a sliding window over the image or tiling it? Then are you marking each pixel as belonging to the edge of a document or are you marking detected edges as valid document boundaries? Also how do you model the links between the 4 sides? A reference to a paper or follow up blog post would be greatly appreciated.

Great work. Laurent