My report was entirely unacknowledged along with other reports that had negative findings. The team in charge published a self-report about the success rate and claimed over 90% perfect results.
About a year later, upper management changed to this style of hard requiring LLM usage. To the point of associating LLM api calls from your intellij instance with the git branch you were on and requiring 50% llm usage on a per-pr basis otherwise you would be pip-ed.
This is abusive behavior aimed at generating a positive response the c suite can give to the board.
That said, I don't agree with or advocate the specific rollout methodology your company is using and agree that it feels more abusive and adversarial than helpful. That approach will certainly risk backfiring, even if they aren't wrong about the large-scale usefulness of the tools.
I wasn't sanctimonious to you, don't be so to me please.
> you would genuinely need to
> look at the full dataset that
> team collected to draw any
> meaningful conclusion here
I compared notes with a couple friends on other teams and it was the same for each one. Yes it's anecdotes but when the same exact people that are producing/integrating the service are also grading its success AND combine this very argument while hiding any data that could be used against them, I know I am dealing with people who will not tell the truth about what the data actually says.
It is hardly feasible for an organization to budget time for replicating and validating results, form their own conclusions, for any employee form who wishes to question the effectiveness of the tool or the manner of deployment.
Presumably the organization has done that validation with reasonably sized sample of similar roles over significant period of time. It doesn't matter though, it would be also sound reasoning for leadership to take a strategic call even when such tests are not conducted or not applicable.
There are costs and time associated with accurate validation which they are unable / unwilling to wait or even pay for, even if they wish to. The competition is moving faster and not waiting, so deploying now rather than wait and validate is not necessarily even a poor decision.
---
Having said that, they can articulate their intent better than "write about how it made you more productive", by adding more description along the lines of "if not then explain all the things you have tried to try and adopt the tool and what and how it did not go well for you/ your role"
Typically well structured organizations with in-house I/O psychologists would add this kind of additional language in the feedback tooling, line managers may not be as well trained to articulate it in informal conversations, which is whole different kind of problem.