Re:The Design and Use of QuickCheck
by alexwlchan on Tuesday June 13, 2017 @02:06AM
(#14543112)
For CI systems like Travis, people add it to the cached directories, and it's shared between runs. I know Travis, Circle and AppVeyor all have some way to cache data between runs – nominally for dependencies, but .hypothesis works too.
According to our docs (http://hypothesis.readthedocs.io/en/latest/database.html?hig...), you can check the examples DB into a VCS and it handles merges, deletes, etc. I don't know anybody who actually does this, and I've never looked at the code for handling the examples database, so I have no idea how (well) this works.
If tests do throw up a particularly interesting and unusual example, we recommend explicitly adding it to the tests with an `@example` decorator, which causes us to retest that value every time. Easier to find on a code read, and won't be lost if the database goes away.
(Disclaimer: I'm a Hypothesis maintainer)
I think the default storage format Hypothesis uses is a flat file with a diff-friendly format so it's easy for developers to check it into source control, and it's easy for patches to update the database without exploding the git repo size due to giant binary diffs. Sqlite3 might also be an option but I'm not up to date on the details. As a neat side effect of the diff-friendly format, it's easy to review new falsifying inputs added to the database in pull requests.
> a diff-friendly format ... it's easy for patches to update the database without exploding the git repo size due to giant binary diffs
Interesting - I understood that Git stores whole files, not diffs, so I'm surprised this is a significant feature.
Of course, with randomised testing, there's an inherent non-reproducibility, so maybe this isn't as unfortunate as it sounds?