I think the confusion might come from the Calorie/calorie difference, i.e. 1 Calorie = 1000 calories = 1 kcal.
We are still much more efficient mind, 2400kcal is approx 2.7kWh so about 1000 GPT queries if your 2.9Wh figure is correct (wow!)
With nuclear power and a way to scale up we could build a gigawatt tier brain if we wanted to.
And in the end, this long term predictions based on where we are now and how fast we are moving are always a bit iffy. Nobody really know what challenges must current AI overcome to become AGI (if that even means anything).
For the short term it's honestly way harder to say, there could easily be another 50 year winter in between. But machines improve, and humans don't, so as long as non-zero progress is made it's a virtual certainty.
It's hard to see a value in humanity when most of the globe is a giant dumpster fire. We currently have:
1. United States, a corporate-owned two-party clusterfuck, potentially on a brink of failing into an autocracy. Because the world is in post-ironic phase, this is all turned into a TV show for the mass entertainment. 2. Russia, a kleptocracy turned into an authoritarian police state run by an ex-KGB ex-Stasi nazi. Always messed up, badly, waging wars as a hobby. Doing their best to fuck up things abroad so they aren't alone, and fuck things up domestically so people are more obedient (dictatorship 101: people always behave "better", when their base needs are just barely satisfied so their focus is all on survival). 3. Middle East, where all things are so fucked up beyond recognition I don't even... 4. China, another anti-utopia lover's dream, with obligatory facade of "harmony" and "prosperity". Don't know much about their internal affairs, to be honest, but don't see any good coming out of it. 5. Africa isn't doing any better, we just hear and care less about their mess here because we have our own mess at home. 6. A bunch of ticking bombs (pollution, global warming, overpopulation crises) so we won't get bored by the old wars alone.
And all of this stems up from the fundamental issue of human nature. Innate high susceptibility to things that sound nice (aka populism) leading up to blind faith, constantly abused by high achiever sociopaths, toppling societies left and right despite all the checks and balances, wreaking havoc. We know about logical fallacies, but still - en masse/at scale - fail at those, badly. We only realize the writings were on the wall all the time, when we're already stuck deep in the mess we've made ourselves.
The worst part of it? There seems to be no solution. At least not that I've ever heard of. Pandemics showed us that a global challenge doesn't reunite people but rather gets abused to divides them further, so even evil aliens' arrival or GAI-gone-Skynet most likely won't change anything.
Our best chances are achieving global post-scarcity and/or solving death (so, yeah, very realistic goals straight outta sci-fi books /s), but not even those are guaranteed to work - just a slim chance that maybe if those major issues are out of the picture something will change and maybe there will be a slim chance it won't get abused for worse. Transhumanism might be another chance (if our monkey brains are a problem, maybe we can patch them up or possibly hop over to something else) but I have my doubts.
What's to value? That humans have coping mechanisms that lets them just ignore all the crap going around and live happily in their imaginary safety bubbles, until it's their time and the frog is finally notices it's being boiled?
(Please don't get me wrong, zero hate here - except the hate for all the abusers and hate for all the mess happening. I love humans, but I'm deeply saddened that something is really wrong with us that we just can't get our shit straight for multiple millennia, stepping on the same rake over and over.)
Yea, I guess I'm deeply unhappy.
Now, I actually do not believe you are right and I am more inclined to believe that artificial intelligence stemming from humanity would see the value in human ideals. Certain things like love, morality, and humility have been deeply held ideals for millennia. I would suppose that an AI would, to some extent, value these things too. Think more “The Day the Earth Stood Still” than Terminator.
This is all conjecture, but one thing I would suggest to you: if you see the natural evolution of your own species to be the removal of itself, you probably should be nowhere near AI or alignment. What you suggest is akin to world-wide genocide. Ironically, in my estimation, you are less aware of humanity than AI will be.
The way I imagine it you could talk to one of these things and it would appear human, except it would know everything that is known, be a master of everything that can be done, and could fax itself to Mars over a laser in 20 minutes. Being software is such a huge advantage, since you can just swap your body for whatever you need to do or wherever you need to be.
I'm sure humans will still be around long into the future, but they won't be doing anything notable because we won't be capable of thinking on a cutting edge level. Not entirely unlike what a cat or a dog is to us now. The inteligence disparity will probably be similar anyway.
I see this as a good thing though, the only thing that separates us from everything else in the universe is our inteligence, in some sense we are that inteligence and this is the next evolutionary step it needs to finally break out from the confines of a tiny 20W brain that's been pushed as far as it practically can be.
When you no longer need to consume energy constantly unless you die, are functionally immortal and can hibernate for a thousand years, you eat only sunlight, don't get cancer from radiation or bone loss from zero gravity... the galaxy becomes your oyster.
Evolution, Morpheus, evolution. Like the dinosaur ;)