I’m not sure why you’d need or want a randomised controlled trial to determine the colour of the sky. There have been empirical studies done to determine the colour and the reasoning for it - https://acp.copernicus.org/articles/23/14829/2023/acp-23-148... is an interesting read.

If you point a spectrometer at the sky during the day in non-cloudy conditions you will observe readings peaking in the roughly 450-495 nanometers range, which crazily enough, is the definition of the colour blue [0]!

Then you can research Rayleigh scattering, of which consists of a large body of academic research not just confirming that the sky is blue, but also why.

But hey, if you want to claim the sky is red because you feel like it is, go ahead. Most people won't take you seriously just like they don't take similar claims about AI seriously.

[0] https://scied.ucar.edu/image/wavelength-blue-and-red-light-i...

you needed a spectrometer to tell you the sky is blue?

Ever seen a picture of the blue sky from the ISS?

I can see it, it's independently verifiable by others, and it's measurable

The same is true of AI productivity

https://resources.github.com/learn/pathways/copilot/essentia...

https://www.anthropic.com/research/how-ai-is-transforming-wo...

https://www.mckinsey.com/capabilities/tech-and-ai/our-insigh...

lol those are all self-reports of vibes

then they put the vibes on a graph, which presumably transforms them into data

"Both GitHub and outside researchers have observed positive impact in controlled experiments and field studies where Copilot has conferred:

55% faster task completion using predictive text

Quality improvements across 8 dimensions (e.g. readability, error-free, maintainability)

50% faster time-to-merge"

how is time-to-merge a vibe?

The subject is productivity. Time to merge is as useful metric as Lines of Code to determine productivity. I can merge 100s of changes but if they are low quality or incur bugs, then it's not really more productive.

this guy has elsewhere in this thread cited "a16z revenue benchmarks" as evidence of productivity. you know, the sector most famous for setting more money on fire faster than anyone in living memory.

> https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

Shows that devs overestimate the impact of LLMs on their productivity. They believe they get faster when they take more time.

Since Anthropic, GitHub are fair game here’s one from Code Rabbit - https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-gen...