It's usually called vexatious litigation, and many jurisdictions have list of vexatious litigants, whose filings go through a special judicial review before being accepted into the system:

https://en.wikipedia.org/wiki/Vexatious_litigation