Step 1: We are going to estimate tasks in imaginary relative units
Step 2: We are now going to use these imaginary units to measure your velocity
Step 3: We are now going to compare your progress in imaginary units to other teams who use a different imaginary units, and complain your underperforming
Then you imaginary unit inflation and before you know everything is an XXXXXXXXXXXXXXXL t shirt size.
The same people are responsible for the estimate as the delivery. The obvious maximum solution for a team is to continually inflate all estimates to demonstrate improved velocity. At the individual level, it is to always take the most over-inflated story in order to demonstrate maximum individual contribution.
You get perverse outcomes with all metric based productivity measures (for example in retail sales having friends buy large volumes and return to another store). You can prevent this kind of thing to a degree through various sticks, but with “planning poker” it is the antithesis of the desired goals. Measure in a different way.
But story points themselves are actually guesses, and are never a true reflection of the work undertaken post mortem - and rarely if ever get reevaluated with 'correct values' mainly because it's really difficult to do that - it's another guess really.
I think any attempts to measure the accuracy of guesses on imaginary relative units, specific to a team, with deliberately obfuscated and difficult to reconcile units is futility by definition.
Instead companies have started to see the light of measurement using standard units of time and tracking the start to finish time including the states of tickets - Flow metrics.
Which is all fine, however you still have the original problem - unless you strictly compare very similar tasks perhaps done by similar engineers in terms of tenure, skill level and maturity, and maybe other factors, you still can't directly compare tasks across teams.
There still needs to be consideration of the make up of a team to get an understanding of skill level and experience.
Ive personally witnessed teams with graduates in them and people with 20 years experience completing very similar tasks and the difference is astonishing.
The big-picture view is given by Exhibit 1, and you can see it's really much more about analyzing all aspects of productivity (company-level, team-level, etc.) to identify bottlenecks around e.g. too much time spent testing over writing code, or senior developers stuck in endless cross-team alignment meetings which means management needs to identify a single person responsible for making final decisions quickly. So this is a document for engineering managers, not engineers.
But in terms of measuring actual individual developer productivity, if you read through all the verbiage it's really mostly just adding up the "planning poker" agile/sprint points that a developer delivers over the course of months. About which I have two main thoughts:
1) As long as planning poker is executed correctly -- i.e. always with unanimous consensus from engineers and without managers/PM's providing pressure -- this is actually probably going to be pretty decently objective, as a moving average. It's not terrible -- it's infinitely better than things like "lines of code". Although teams can also experience gradual "points drift" over the course of months, and points between teams can't be translated, so it can't be used company-wide or even year-to-year. But it will compare productivity between team members.
2) BUT, there's a huge risk involved because not all e.g. 5 point stories are the same, because of variance. Some are very straightforward where you know it's 5 points, while others are much more unknown, where the best guess is 5 points -- and it might wind up being 2 points but it also might wind up being 50 points. And so I could definitely see certain stories turning into "hot potatoes" nobody wants to touch because it's just too much of a risk -- it'll tank their average and they won't get promoted. And then everything just breaks down. Which means you either need post-sprint processes for "correcting" estimates and assigning a reasonable value for points done, or going deeper into "research" stories -- e.g. a 1-point pre-story to better nail down the actual size.