You are 100% correct about what could have been. However, the reality is that Einstein's work paved the way for Leo Szilard to envision an atomic bomb before fission had been discovered.

http://nuclearweaponarchive.org/Usa/Med/Discfiss.html

Edit: Reflecting on this, without quantum mechanics (which stemmed from Einstein's relativity work), Oppenheimer would not have had the mathematical tools that were heavily leveraged building the bomb. The bomb is more than throwing together two sub-critical pieces of material, knowledge from quantum mechanics was critical in the prediction of which fissile materials could yield a functional weapon (of which there are very few).