I was confused, how did he figure it out? Is it somewhere I missed? I checked the query in the dashboard and it was this insane looking query that was correctly looking for the shared userAgent, but also doing a shit ton of odd functions and weird looking math. I asked the guy and he pointed me to the obligatory 1,400 line markdown file describing the “plan” and “approach”. In between all the raw LLM-isms and very sophisticated “weighted normalized distribution of language spread over time” sounding sentences, there was a sentence that said “Since we only have one userAgent for all tools, we use industry standard distribution of language popularity to approximate the usage, and apply some randomness to emulate an organic usage pattern”
So basically the AI just made up some numbers that look plausible (python 20%, JavaScript 30%, CLI 25%, etc) with some randomness thrown in “to look organic”, and the guy was like “that checks out”. When I pointed that these numbers are wrong because X, Y and Z, the person who asked for the data said “that’s ok, it’s a great start and gives me something to use for planning with other teams. Feel free to refine them if you want, but it’s good we have some numbers”. I’m planning my exit soon.
"Then this product is not for you." Right, not for anyone thinking critically about how any of this works.
I guess my time zone wasn’t in the context? So it just hallucinates the wrong coastal time zone twice when I’m not in either one. But who knows where exactly it messed up because it could have just picked a random hour and a would have had the same outcome.
I’ve only used Alexa a half dozen times since the release of Alexa+ but it has been confidently incorrect about 100% of the queries.