People essentially spent entire careers trying stuff, until comparatively recently when materials science theoretical modelling became good enough.

The classic Ignition! is well worth reading to see the process from the inside. The basic figure of merit for rocket fuels, specific impulse, could be calculated on paper. All sorts of other critical properties like detonation resistance had to be found experimentally. As well as the trick to storing red fuming nitric acid without eating through stainless steel.