the only barriers to understanding the code is power and time. are you seriously advocating for an unintelligible object, made by our own hand? that - in itself - is considered intelligent?
someone's going to get burnt trying to chase two meanings at once, and it's not going to me, i'm sorry.
if you want to discuss centuries-long innovation, then the counter-argument is just more centuries of interpretation and dissection to understand it. if you want to guesstimate AGI arriving at 2220 then the balancing counter-technology comes in at 2420.
people are already starting the process of cracking open the black box. [0] the majority of what keeps AI and any software "in the dark" is the cloud's cybersecurity and legal mechanisms (and the need for man-power) that prevent the average researcher breaking into it.
it is still vulnerable to the scientific method and symbolic interpretation. as always.
LLM's main advantage is keeping knowledge out of the hands of the poor and the public. The millisecond AI becomes "good enough" people will use it as a cover-up justification to start hiding currently accessible knowledge behind paywalls and university entrance exams.
AI is the grim reaper for the world we used to know.
[0]https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti...
it's actually very hard to imagine. i don't think you realize how little we understand the things you claim a machine can't have.
We are at best decades away (maybe centuries) from codifying intelligence as you imagine. why do you think llms or even just machine learning in general are the revolution they are ? We found a way to create intelligence without figuring out the hard part. as a result, the creations often become as black box as their creators.