How do you know that they don't just flat out lie about these files being downloaded by your IP?

In Poland company bought out rights to one porn movie, then claimed to the police that that file was downloaded by random ip addresses (out of major providers pools) on torrent. Police ordered the companies to release home addresses of those ip users. Then the company slent email demanding settlment money. Some people paid. Now they are suing the company to get their money back. So many people police about this that they issue public statement that there was nothing criminal and they dropped the case.