Probably. If you look at the sensitivity of passenger traffic to travel time, the graph is roughly a logistic curve with the inflection point around the 3 hour mark. On a 40-hour trip, the number of people you'd add to the train by cutting out 4 hours of stops is probably on the order of 10-100 people per trip. If those stations collectively serve more than that number of people, then you're losing people by cutting those stops.