David Hume arrived at a reasonably rigorous framework (Principle of Nature's Uniformity) that allows your conclusion, in my opinion. He's worth reading, if only to find enough closure to move on from philosophy.