You're not wrong. What people think will depend on context, and probably on region. In US English, jacket is a wide ranging term that could easily include a zip up sweater. If you wanted to be extra clear, you could saw "sweater jacket" or "fleece jacket" (if it's not woven).