News for hackers. Stuff that matters.
Re:Property-Based Testing for the People by rtpg on Monday January 06, 2025 @05:49PM (#42616970)
I understand your pain in some sense, but on another I feel like people with a decent amount of hypothesis experience "know" how the generator works and would understand that you basically _never_ want to use `filter` if you can avoid it, instead relying on unfalsifiable generation.

Silly idea for your generator would to generate an array, and if it's zero... draw a random index and a random non-zero number and add it into the array. Leads to some weird non-convexity properties but is a workable hack.

In your own example you turned off the "data too slow" issue, probably because building up a dataframe (all to just do a column sum!) is actually kind of costly at large numbers! Your complaint is probably actually meant for the pandas extras (or pandas itself) rather than the concept of hypothesis.

Re:Property-Based Testing for the People by epgui on Monday January 06, 2025 @06:53PM (#42617539)
No, I ran into the same issues with basic data structures. The dataframe wasn’t necessary, it just matched the expected input of some function I wanted to test.
Re:Property-Based Testing for the People by rtpg on Monday January 06, 2025 @08:14PM (#42618129)
I took your case, I got way better perf just generating a list of numbers and then reshaping it into a dataframe.

But! Even though it doesn't even get that much slower at a certain number of rows it just starts hanging! Like at 49 rows everything is still fine and at 50 it no longer wants to work. It's very bizarre and I'll see if I can debug it. But I think your test case isn't indicative of some fundamental issue with Hypothesis rather than some sort of bug.

Re:Property-Based Testing for the People by tybug on Monday January 06, 2025 @08:46PM (#42618302)
That kind of behavior can happen at the threshold of Hypothesis' internal limit on entropy - though if you're not hitting HealthCheck.data_too_large then this seems unlikely.

Let me know if you have a reproducer, I'd be curious to take a look.

Re:Property-Based Testing for the People by rtpg on Tuesday January 07, 2025 @12:14AM (#42619489)
yeah seems like I had pulled in the "too large" healthcheck supression.

The thing I don't get is that if I do disable it, then there's just an arbitrary failure later down the stack, but maybe that error can be more explicit? Or is this a bit unknowable when you run out of entropy?

Like why is hypothesis not throwing an `OutOfEntropy` error akin to an OOM? It _is_ crashing

Re:Property-Based Testing for the People by epgui on Tuesday January 07, 2025 @02:42PM (#42626487)
> Even though it doesn't even get that much slower at a certain number of rows it just starts hanging

Yes, this brings back memories. I've definitely seen this kind of behaviour as well, in many different, not-particularly-exotic, situations.

I am absolutely convinced the issue I raised on the github project was a bug or a defect, despite the maintainers not taking it seriously.

I find QuickCheck and Clojure spec/test.check much more straightforward to use. I just never ran into this sort of thing with these other tools.