The term open source implies the code is free. Simply having the source available does not make it open source.

And how would you enforce that? What would prevent anyone with access to the code from building it and using it? I don't see any way except maybe stripping the code of significant parts

You could keep the virus fingerprint database outside the codebase. Customers would then pay for access (and updates) to the fingerprints.

The fingerprints have to be some sorts of data, like regular expressions or other limited instruction set which can only parse the incoming file and not communicate with outside world.

The company could automatically release fingerprints into the open after a time, say 6 months.

So... stripping the code of significant parts

This seems like a pretty good business model idea though!

Copyright laws?