Your transoceanic cable example is an interesting one, as the first electrical / telegraph cables greatly expanded the understanding of electric fields and interactions with the environment, especially in salt-water.
I'm also wondering if there's some sort of frontier between the "just blundering around" approach --- mass parallel experimentation --- and "requires a substantive theoretical understanding". To take your LED example, LEDs are the inverse of the photoelectric effect (and apparent PV panels will emit photons when a charge is applied to them). Electroluminescence dates to 1907, whilst the first LEDs were developed in the 1960s. There were earlier similar phenomena such as chemoluminescence (including numerous examples of bioluminescence) which might have suggested the possibility.
I'm agreeing in part, disagreeing in part, and wondering if there might be a more robust or systematic way of distinguishing limits of both methods.
Sometimes it's surprising what nobody was interested in inventing. I think Grace Hopper is really important because people were resistant to the idea that programming the computer involved boring mindless steps which could be done by a machine and so of course instead of hand writing the program in machine code you should write a higher level language and have the machine translate that. It's incredible now, but this very idea was once an important invention and yet her superiors were not enthusiastic.
Bernhard J. Stein's *Resistances to the Adoption of Technological Innovations" (1937) is a fascinating read in this regard:
https://archive.org/details/technologicaltre1937unitrich/pag...
As Markdown: https://rentry.co/szi3g
I'd heard of it via Isaac Asimov who mentions it in his biography and a few other contexts. Asimov was Stern's research assistant, and incorporated the ideas into several of his own stories.
You can discover that eating one of the dozen types of plants growing on the hillside nearby treats toothache without any overarching theory about how that could work, just try eating stuff and see what happens - but you aren't going to invent the LED lamp this way.
Example: When we put a cable on the bottom of the ocean these days it's optical fibre rather than electrical. But, even with optical fibre, even the best stuff we can make, this will need amplifying for long distances or it's pretty awful. One thing you could do would be to choose reconstructing amplifiers when making the cable. So e.g. you decide this cable is Protocol X at 100Gb/s, you make amplifiers which can reconstruct a Protocol X signal at 100Gb/s and "boost" it, splice those in along the distance of the cable, and drop the whole lot into the ocean. However, somebody is going to invent 500Gb/s Protocol X+ and if you want to upgrade you will need to send teams down to the ocean floor to replace those amplifiers. Ouch.
In principle individual photons are travelling along the fibre, and physics doesn't say we can't just have one photon in => two photons out to boost this without needing to reconstruct the signal at all. There should be some way to build a device which does this, an Optical Amplifier, and it would be OK if this is quite expensive since it's saving you that enormous expense by allowing you to upgrade to 500 Gb/s X+ or to 10Tb/s XXX or whatever other future protocols just involve sending photons down a fibre without trying to upgrade equipment at the bottom of the ocean. But... how?
Turns out scientists can guess exactly how that should work if it's possible, and then direct the experiments, trying out only the handful of things which actually might work instead of just groping about at random. My alma mater was one of the places figuring out how to do this in the 1980s, they were still really proud of that when I studied there a decade later. Erbium Doped Fibre Amplifiers are the result.