If management believes doing either one of those will result in a financial benefit greater than paying the dividend/doing buybacks, they will do so.

The idea is that if a company doesn't have compelling business plans requiring further investment that will likely generate a significantly better ROI than the cash can generate, then the excess cash should be returned to the owners of the company. (i.e. the shareholders) A secondary benefit is that dividends make stocks more expensive to short (i.e. speculators betting the share price will go down... and often helping it do so) and can often increase the price paid in hostile takeovers (i.e. the acquirer will often have to pay a higher premium to get the shareholders to accept than if the company just kept the money in cash equivalent investments)

What has changed in recent decades is that dividends are generally not considered the most tax-efficient way to get the money to the shareholders so stock buybacks (which presumably increase the share price and will be taxed as capital gains when shareholders sell as opposed to income when received as dividends) are often preferred.