Maybe those riders live on that street, or need to use the larger road briefly to access a turn that isn't safe to access via the bike lane? Or maybe they're riding faster than is comfortable on a shared path (kudos to them -- racing is not safe in an environment shared with pedestrians). Bicycles have every right to the road that cars do, so those bikes are perfectly entitled to use that road. Do they really slow down traffic that much? If that stretch of road has traffic lights, it probably doesn't make much of a difference.