Most spices were expensive because distance and perishing. Faster travel and refrigeration fixed that.

Saffron is expensive because takes an insanely large volume of arable land and manual labor to produce. A saffron crocus plant flowers once per year. It produces about four flowers, and each has three tiny threads of saffron. They are harvested by hand.

To produce a pound of saffron requires about 50,000-70,000 flowers and 40 hours of manual labor.

It is very expensive relative to other foods, and certain varieties of Persian saffron are extremely expensive.

It is also extremely inefficiently used by most chefs. They often merely crumble it into the recipe. You can use a small amount of saffron to flavor a large quantity of food simply by grinding it into powder and mixing with a small amount boiling hot water. The resulting mixture is very potent.

To be fair saffron is the stamen and stylr of Crocus sativus. There's only a few per flower and they only bloom in Autumn.