Re:Using achievement stats to estimate sales on Steam
by bartervg on Wednesday July 04, 2018 @06:39AM
(#17456377)
Note the addition to the end of the post,
UPDATE (7/4): Looks like valve is rounding numbers on the API now so this method no longer works.
I am not the author of the Gamasutra blog, but I have worked on the estimating method. It used Steam's public API [1] and within the last few hours, the effective precision was reduced to 1 decimal.
Potentially related, Steam's stated problems with Steamspy [2]
1. GDPR
2. Broad variance of accuracy
Re:Using achievement stats to estimate sales on Steam
by larsiusprime on Wednesday July 04, 2018 @07:23PM
(#17459959)
It was already in public use for several weeks before and developers had tweeted publicly about it. At worst the blog post accelerated the nerfing of the API, but it was inevitable once publicly disclosed.
Re:Using achievement stats to estimate sales on Steam
by hhmc on Wednesday July 04, 2018 @06:41AM
(#17456383)
It's unsurprising this was patched out shortly after being made public. Despite the truncated rounding, the technique could probably still be valuable. Sampling multiple points in time, then analyzing the common multiples each time point yields to give a likely range.
This requires that the achievement percentages vary with time (which could be a flawed assumption).
Re:Using achievement stats to estimate sales on Steam
by fenomas on Wednesday July 04, 2018 @09:52PM
(#17460427)
> Despite the truncated rounding, the technique could probably still be valuable.
How would that work exactly?
As long as a title has over, say, 10,000 sales, rounded percentages don't eliminate any possible values for the total number. If one value doesn't give you any information, how would sampling more values help?
Re:Using achievement stats to estimate sales on Steam
by vermilingua on Wednesday July 04, 2018 @07:31AM
(#17456594)
“Why would it hurt them” and “Why can’t Valve leak potentially sensitive financial details of their partners/clients” are two ways of wording the same question here.
Re:Using achievement stats to estimate sales on Steam
by shoo on Wednesday July 04, 2018 @07:31AM
(#17456598)
it lets people accurately estimate sales / revenue of businesses, or market demand. that's valuable information that game developers / publishers / steams might not want to share by default without getting something valuable in return.
Re:Using achievement stats to estimate sales on Steam
by CamTin on Wednesday July 04, 2018 @01:02PM
(#17458270)
They must have reasons for not wanting this figure to be public. If they didn't, they would just expose it directly via the API or in the UI instead of making people solve a fun math puzzle to tease it out of other figures.
Re:Using achievement stats to estimate sales on Steam
by derefr on Wednesday July 04, 2018 @02:23PM
(#17458712)
If they're the only one with the dataset, they can sell that dataset to publishers. If someone else scrapes the dataset from them, though, then that person or organization can then also sell it to publishers, reducing its market value when Valve goes to sell it.
This is the same reason LinkedIn makes it so hard to scrape their graph.
Re:Using achievement stats to estimate sales on Steam
by CamTin on Wednesday July 04, 2018 @02:27PM
(#17458731)
Sure, that's one possible reason. Whatever their actual motivations, one thing that we know for sure is that the don't want this data out there. Therefore it should be no surprise that they dull the accuracy on achievements as soon as it is reported that this can be used to divine the numbers. There's not a logically-consistent position Valve could have where they both a) don't want to provide the data directly, but are b) totally fine with you determining it with extreme precision based on other available data. Such a Valve does not and cannot exist.
Re:Using achievement stats to estimate sales on Steam
by larsiusprime on Wednesday July 04, 2018 @07:25PM
(#17459962)
I know tons of developers and publishers and I have never ever heard of Valve selling their dataset to publishers. Also based on the TOS that I signed with them as a developer (and presumably is the same for every other developer) it would be a violation of their contract with me to share my stats without my permission.
Re:Using achievement stats to estimate sales on Steam
by phobosdeimos on Wednesday July 04, 2018 @05:15PM
(#17459479)
Publishers and developers don't want sales numbers public because gamers will quickly pick up on low sales and meme you to death. Any bad exposure on the internet can sink a game real fast.
Re:Using achievement stats to estimate sales on Steam
by anonnyj on Wednesday July 04, 2018 @09:08AM
(#17457046)
Random thought about the accuracy: If you published a game on Steam, you've probably noticed that many owners haven't cleared even "opened up the game" tier achievements.
Re:Using achievement stats to estimate sales on Steam
by bartervg on Wednesday July 04, 2018 @10:47PM
(#17460551)
Yes, this is an issue.
One example is FreeFly Burning, which according to the achievement estimation method has 739 players [1]. Yet, the game has sold 25,562 keys through Go Go Bundles, and more through other bundles. Go Go Bundles lists the number of bundles delivered [2] and each bundle includes every game (in contrast to Humble Bundle that has tiers). This is consistent with Steamspy's estimate of 20,000 - 50,000 owners [3]. Those bundle buyers likely never downloaded the game and perhaps haven't even activated the game on Steam.
With exceptions like F2P games or free weekend games:
Units Sold >= Owners >= Players
Re:Using achievement stats to estimate sales on Steam
by larsiusprime on Wednesday July 04, 2018 @03:08PM
(#17458925)
I reviewed the algorithm, and as long as you have SOME users at all registering some amount of achievements, the method is quite accurate. Even if most players register no achievements, they still count towards the overall pool of "achievers" and provide a solid base for doing the calculation.
You can verify this yourself with the supplied source code.
The only cases where it can fail are:
1) When there is no achievement data
2) When there are not enough independent achievement %'s.
2 Can happen for games with very few achievements -- but in practice, if a game has even 3 achievements the chances of this happening become very small.
TL;DR -- for very nearly every game on Steam, this was an extremely accurate method for estimating the number of "achievers" for that game. The tricky bit is correlating "achievers" with a metric you actually care about, like e.g. actual sales.
Re:Using achievement stats to estimate sales on Steam
by larsiusprime on Wednesday July 04, 2018 @07:30PM
(#17459975)
Yes you do. Because even if they don't register an achievement, they still show up as part of the total population of possible "achievers". I've verified this from my own stats as a developer and that of other developers.
As long as any portion of the userbase registers some achievement data, you can (well, could) get scarily accurate stats.
The only "inaccurate" part is that the stat you're measuring -- "achievers" -- doesn't exactly correlate to any stat we directly care about, like sales. In all the measuring I did against known private sales data, it was always higher than the # of people who had downloaded the game, and the # of people who had played it for at least 1 minute. My best guess is family sharing inflates the "achievers" stat.
Re:Using achievement stats to estimate sales on Steam
by pixl97 on Wednesday July 04, 2018 @09:53AM
(#17457251)
I've bought many games on sale and never installed them. Maybe one day I will, maybe I wont. My take is games that never have sales are more likely to be installed and ran.
Re:Using achievement stats to estimate sales on Steam
by GuB-42 on Wednesday July 04, 2018 @10:24AM
(#17457416)
Also, bundles.
I have a dozen of game I have never played, and will probably never play. They were part of a bundle that was cheaper than the individual game(s) I wanted.
Re:Using achievement stats to estimate sales on Steam
by hhmc on Wednesday July 04, 2018 @01:22PM
(#17458376)
Also, some games had their achievements added after release (in some cases years). Achievements don't usually get backfilled to my knowledge, so any players with playtime before achievements will be underrepresented.
Re:Using achievement stats to estimate sales on Steam
by larsiusprime on Wednesday July 04, 2018 @07:45PM
(#17460019)
This isn't the case, actually. Testing it against known sales data, the "achievers" stat derived by this method was ALWAYS higher than the # of ppl who had downloaded the game, or the # of ppl who had played it for at least 1 minute (these stats are known privately to developers).
We don't know exactly how the total "achievers" pool for a game is calculated, but we know it isn't a simple measure of people who had achieved at least one achievement. It's almost certainly some measure of the total population that is able to register achievements, regardless of whether they have registered any.
It probably includes something like family sharing accounts, so it doesn't exactly correspond to actual sales directly, but it seems to be roughly in the middle between the "players" and "owners" stats SteamSpy used to report. Sometimes it would overshoot both, but it was never less than "players."
Re:Using achievement stats to estimate sales on Steam
by AdmiralAsshat on Wednesday July 04, 2018 @03:09PM
(#17458937)
Yep. I even wonder if Steam has some kind of counter that would show me "Number of games you own that you've never played." I would probably be horrified.
It splits out a chart showing you how many games you have in a few different playtime categories ("12 or more hours", "6 to 12 hours", "Never played", etc.)
I am not the author of the Gamasutra blog, but I have worked on the estimating method. It used Steam's public API [1] and within the last few hours, the effective precision was reduced to 1 decimal.
Potentially related, Steam's stated problems with Steamspy [2] 1. GDPR 2. Broad variance of accuracy
[1] https://developer.valvesoftware.com/wiki/Steam_Web_API#GetPl... [2] https://twitter.com/GamesCodeDogs/status/1012265304771252224
This requires that the achievement percentages vary with time (which could be a flawed assumption).
[1] https://steamcommunity.com/stats/Portal2/achievements
How would that work exactly?
As long as a title has over, say, 10,000 sales, rounded percentages don't eliminate any possible values for the total number. If one value doesn't give you any information, how would sampling more values help?
And Twitter recently now lets you see the ads for any company/competitor as well. its not sales figures but its still competitor intelligence.
If you're one of the top 10k products within a main category Amazon will release it in their public data anyway though.
This is the same reason LinkedIn makes it so hard to scrape their graph.
One example is FreeFly Burning, which according to the achievement estimation method has 739 players [1]. Yet, the game has sold 25,562 keys through Go Go Bundles, and more through other bundles. Go Go Bundles lists the number of bundles delivered [2] and each bundle includes every game (in contrast to Humble Bundle that has tiers). This is consistent with Steamspy's estimate of 20,000 - 50,000 owners [3]. Those bundle buyers likely never downloaded the game and perhaps haven't even activated the game on Steam.
With exceptions like F2P games or free weekend games: Units Sold >= Owners >= Players
[1] https://barter.vg/i/40567/a/
[2] https://www.gogobundle.com/latest/bundles/bundlemega-7
[3] https://steamspy.com/app/683530
You can verify this yourself with the supplied source code.
The only cases where it can fail are:
1) When there is no achievement data
2) When there are not enough independent achievement %'s.
2 Can happen for games with very few achievements -- but in practice, if a game has even 3 achievements the chances of this happening become very small.
TL;DR -- for very nearly every game on Steam, this was an extremely accurate method for estimating the number of "achievers" for that game. The tricky bit is correlating "achievers" with a metric you actually care about, like e.g. actual sales.
As long as any portion of the userbase registers some achievement data, you can (well, could) get scarily accurate stats.
The only "inaccurate" part is that the stat you're measuring -- "achievers" -- doesn't exactly correlate to any stat we directly care about, like sales. In all the measuring I did against known private sales data, it was always higher than the # of people who had downloaded the game, and the # of people who had played it for at least 1 minute. My best guess is family sharing inflates the "achievers" stat.
I have a dozen of game I have never played, and will probably never play. They were part of a bundle that was cheaper than the individual game(s) I wanted.
We don't know exactly how the total "achievers" pool for a game is calculated, but we know it isn't a simple measure of people who had achieved at least one achievement. It's almost certainly some measure of the total population that is able to register achievements, regardless of whether they have registered any.
It probably includes something like family sharing accounts, so it doesn't exactly correspond to actual sales directly, but it seems to be roughly in the middle between the "players" and "owners" stats SteamSpy used to report. Sometimes it would overshoot both, but it was never less than "players."
It splits out a chart showing you how many games you have in a few different playtime categories ("12 or more hours", "6 to 12 hours", "Never played", etc.)