Latter has happened quite a few times in experience. Algorithm performed as designed on the training or example set just to fail terribly on real life data.

Or someone handwaved important things like having an information function be available (impossible, used only in proofs of correctness, way of estimating it is critical).

Or a key assumption on input was just mentioned somewhere in the depths of the paper.

Or a very specific way of measuring the result hides the deficiencies. (Similar to p-hacking or misusing stats in medicine.)