> Interestingly, one of the techniques used by the banks to dupe the rating agencies [before the 2008 economic crisis] was to reverse-engineer the computer models the agencies used to devise their ratings. Reverse-engineering was facilitated by the fact that the rating agencies made their computer models public. Thus the banks could trick them starting from answers and then working backward to get the intended result.
It would seem open-source is not always advisable when it comes to sensitive information.
For behavior models like risk ratings sure. A similar example is threat and fraud detection; if scammers know what you're checking it'd be easier to work around.
In this case though the output is supposed to be a "fair" random number. If knowing how it's generated allows you to be able to predetermine it, i.e. exactly what the offender in this case did, then the code itself is bad and public eyeballs might find it.
Wouldn't hurt to open source it too. It's not a silver bullet as you still need to ensure the build is stable and the final product is actually running that same code, but at least it gives the public something to eyeball. That plus separation of duties (develop/validate/deploy) delegated to separate people is as good as it gets. To get around that you'd need collusion across all the groups and if you've got that, you're totally screwed anyway.