To not lose sight of the discussion subleties, the gp was saying that in this 1st scenario, Bob still didn't "solve it (totally) on his own" because he still depended on the previous work of others to build on. Likewise, we can say Andrew Wiles "solved Fermat's Last Theorem" but Wiles acknowledges that seeing Ken Ribet's proof of epsilon conjecture was a breakthrough he used.
>In another, Carbon-based Charlie breaks into Bob’s lab, copies enough of Bob’s notes to understand Bob’s work, and provides the final insight to solve the open question. In this world, it’s fair to say that Charlie and Bob both contributed to solving the open question, but I think also fair to say that Charlie didn’t have the capability to solve it on his own.
Again, using gp's framing, Bob also didn't have the capability to solve it on his own. By omitting the previous papers and past work that Bob built on, it makes your hypothetical scenario incomplete when judging Bob vs Charlie.
We don't have an objective standard of how much the "standing on the shoulders of giants" applies to each breakthrough. There was a blog post (might have been Terrence Tao) that said society unfortunately awards the fame to the person who solves the last step of a proof and forgets about the people who solved the intermediate steps that led up to it.
Now of course Alice has the advantage of massive parallelization, but that's not a reason to discredit her, that's a genuine advantage that she has, applicable to all problems.
To respond to a couple specific points:
> whether Bob’s contributions were required to Alice’s final step
Even if Bob's contributions were required, I don't think that's a reason to fully discredit Alice.
> Alice is capable of solving a second open math question
Given that AI has already solved dozens of open math questions, I don't think this is up for debate.
We’re not sure whether AI Alice has the capabilities required to definitively solve this open question.
Then, Biological Bob starts diligently working on this problem. He toils and toils, finding many dead ends but a few parts where he makes meaningful progress.
Eventually, Bob knows he’s made some real advances and thinks he might be getting close to the solution and word of this possibility leaks out.
At this point, we all agree that NS is still an open question and neither Alice nor Bob has solved it.
Now, we fork the universe in 3. In one, Bob continues his work and solves the open question (or doesn't).
In another, Carbon-based Charlie breaks into Bob’s lab, copies enough of Bob’s notes to understand Bob’s work, and provides the final insight to solve the open question. In this world, it’s fair to say that Charlie and Bob both contributed to solving the open question, but I think also fair to say that Charlie didn’t have the capability to solve it on his own.
In the third universe, AI Alice does something that you get to define that matches the pattern of facts we know and then we put it to the community to decide whether AI Alice has the capabilities to solve this specific open math problem and whether Bob’s contributions were required to Alice’s final step.
What do you define that she did? What’s the likely community vote on “Alice is capable of solving this specific open math question.” And for those who agree to that, to a follow-up question: “Alice is capable of solving a second open math question.”