You're right! Most castellano speakers don't make a distinction between /b/ and /v/. Some latin american countries do make the distinction and also some spanish regions, like catalan/valenciano speakers in certain regions have a stronger or softer, yet still distinct /v/ sound, such as in "a veure, Vicent" (let's see, Vincent).