Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! I did something like this in college when I was implementing a compiler and it caught a lot of really weird bugs in a lot of peoples codes. The beautiful part of some fuzzers though is they can automatically simplify their complicated test cases to something you can actually look at and reason about.

>Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec!

QuickCheck style testing is maybe also worth a mention here. Instead of using any possible inputs, like in fuzzing, you restrict yourself to legal inputs, like the keywords here, to get maybe less random crashes, but more likely to find useful corner cases because of the restriction on the search space.

SQLite is continuosly fuzz tested by at least 2-3 professional teams[0], I am really amazed you could find 10 bugs in one day. Would love to read your write up in this!

[0]: https://www.sqlite.org/testing.html

so basically, fuzzing.

https://en.m.wikipedia.org/wiki/Fuzzing

Sounds more like smart Monkey testing, much older, less feedback.

https://en.m.wikipedia.org/wiki/Monkey_testing

Edit: looks like some consider these the same nowadays.

I wonder if ChatGPT would be good at generating various fuzz testing queries. Maybe too slow? Maybe too repetitive? Maybe too unvarying? Sorry for mentioning ChatGPT (to those who are sick of the infection).

Actually, with one of the problems being faced in that space at present, especially for applications involving factual answers, being "hallucinations" ( ie. essentially as I understand it the level of "creativity" in responses) such "creativity" may well be quite suited to finding those various unusual edge cases.

You know this common trope in fiction and jokes, that advanced computers & AIs can be easily defeated by feeding them a logical inconsistency?

Guess what, we now have a chatbot AI that not only doesn't mind working with nonsense input, it will happily produce logically-inconsistent statements on its own, and can do it so sneakily and convincingly, that it's the human operator who could end up believing logically inconsistent statements, without even realizing it, and possibly end up in serious trouble some time later.