That would be true if we wanted to convert the carbon dioxide back into carbon, which inherently costs 394 kJ/mol, plus whatever losses you have. But actually we just need to convert it into something that isn't in the atmosphere. My suggestion at https://news.ycombinator.com/item?id=29327736 only requires 178 kJ/mol, much of which can probably be recaptured, and the currently popular scrubber processes using things like triethanolamine require still less energy than that.

It's true that there's an inherent "refrigeration cost" associated with decreasing the entropy in the system. Do you know how to calculate it? How much entropy do you need to shed to concentrate 440 ppm CO2 up to 1,000,000 ppm CO2, and how much energy does that inherently cost? I'm pretty ignorant about thermodynamics.