You have to keep in mind that the way tax collection worked in that area at the time is that it was contracted out by the state. The contractor needed to deliver some amount of money to the state, and empowered to gather money from people; the difference between what they gathered and what they delivered to the state was their profit.

Tax collectors operating within this incentive structure were not likely to be very likable.

And just to drive the point home, the tax policy that this caused was occasionally literally genocidal, in the most strict meaning of the word.

This was possible because it was a slave society, and it was possible for the tax collector to collect your children into slavery in lieu of unpaid taxes. In some areas in Anatolia, which were close to the border of the empire and thus had a significant military presence that the tax collectors could fall back on, so the local population had no possibility to push back, entire societies were ended because the Roman publicans collected all children once they reached the age where they could be profitably sold (circa ~10 years or so, and sold usually to sexual slavery), and did so for long enough that the populations collapsed, never recovered, and were eventually replaced by other populations transplanted from elsewhere in the empire. (Later, the power of the publicani was seriously curtailed, in small parts because even the Romans thought that some of their practices were abhorrent. Albeit mostly because of internal power struggles with the senator class.)

So yes, people in the provinces had plenty of reasons to hate the tax collectors.

Where can we read more about this?

Sort of?

On the one hand, they were a cultural byword for a morally degraded profession at the time. "Tax collectors and prostitutes" is a phrase that comes up not infrequently in the gospels, and not just out of Jesus' mouth So apparently the practice of tax collecting was a little weird in the Roman Empire.

On the other hand, he was known for having dinner with tax collectors and prostitutes. One of his 12 disciples was a tax collector.

Matthew was a publican/tax collector, so perhaps that was written from experience.

In some places, tax collection was performed for an occupying power, e.g., the Roman empire. Hence, it was rightfully seen as "money taken away" instead of "funding public infrastructure".