Unless your product can really benefit from a strong network effect in the open-source community without compromising your competitive edge, I wouldn't do it.
However, if you feel the potential to gain traction outweighs the threat of losing an edge (or a paying customer, since they can just use your code), open-source the engine, but hold back on the parts that are unique to your offering and/or make your engine easy to work with (like a GUI, simplified wrappers or builds for different platforms), though this depends on the nature of your product.
However, non-customers can be a lot of people which would spread the name by word-of-mouth. Also those customers who intend to buy it will get a feeling of what they are getting, instead of you describing a black-box with significant marketing efforts.