Yeah, "factory" is really the general term for a facility that adds value to inputs at an industrialized scale. A lay person would call everything from a smelting plant to an electronics assembly floor a "factory" and not be wrong.

From a pre-industrial blacksmith's perspective, the bigger distinction might be between a "factory" and a "shop". The processes involved are effectively the same; the difference is the scale/flexibility tradeoff (a shop can make different things every day without added overhead, where a "factory" gains enormous efficiency by being configured to do a single process).