News for hackers. Stuff that matters.
Re:Using achievement stats to estimate sales on Steam by bartervg on Wednesday July 04, 2018 @06:39AM (#17456377)
Note the addition to the end of the post, UPDATE (7/4): Looks like valve is rounding numbers on the API now so this method no longer works.

I am not the author of the Gamasutra blog, but I have worked on the estimating method. It used Steam's public API [1] and within the last few hours, the effective precision was reduced to 1 decimal.

Potentially related, Steam's stated problems with Steamspy [2] 1. GDPR 2. Broad variance of accuracy

[1] https://developer.valvesoftware.com/wiki/Steam_Web_API#GetPl... [2] https://twitter.com/GamesCodeDogs/status/1012265304771252224

Re:Using achievement stats to estimate sales on Steam by KaoruAoiShiho on Wednesday July 04, 2018 @04:38PM (#17459335)
Did the blog post ruin the method?
Re:Using achievement stats to estimate sales on Steam by larsiusprime on Wednesday July 04, 2018 @07:23PM (#17459959)
It was already in public use for several weeks before and developers had tweeted publicly about it. At worst the blog post accelerated the nerfing of the API, but it was inevitable once publicly disclosed.
Re:Using achievement stats to estimate sales on Steam by hhmc on Wednesday July 04, 2018 @06:41AM (#17456383)
It's unsurprising this was patched out shortly after being made public. Despite the truncated rounding, the technique could probably still be valuable. Sampling multiple points in time, then analyzing the common multiples each time point yields to give a likely range.

This requires that the achievement percentages vary with time (which could be a flawed assumption).

Re:Using achievement stats to estimate sales on Steam by bartervg on Wednesday July 04, 2018 @07:19AM (#17456538)
It's a valid assumption. For example, Portal 2's achievement Wake Up Call is 67.8% now [1]. Last week, it was 68.8%.

[1] https://steamcommunity.com/stats/Portal2/achievements

Re:Using achievement stats to estimate sales on Steam by fenomas on Wednesday July 04, 2018 @09:52PM (#17460427)
> Despite the truncated rounding, the technique could probably still be valuable.

How would that work exactly?

As long as a title has over, say, 10,000 sales, rounded percentages don't eliminate any possible values for the total number. If one value doesn't give you any information, how would sampling more values help?

Re:Using achievement stats to estimate sales on Steam by AznHisoka on Wednesday July 04, 2018 @07:18AM (#17456533)
Why would they patch it? In other words why would people knowing the sales number of games hurt them?
Re:Using achievement stats to estimate sales on Steam by vermilingua on Wednesday July 04, 2018 @07:31AM (#17456594)
“Why would it hurt them” and “Why can’t Valve leak potentially sensitive financial details of their partners/clients” are two ways of wording the same question here.
Re:Using achievement stats to estimate sales on Steam by shoo on Wednesday July 04, 2018 @07:31AM (#17456598)
it lets people accurately estimate sales / revenue of businesses, or market demand. that's valuable information that game developers / publishers / steams might not want to share by default without getting something valuable in return.
Re:Using achievement stats to estimate sales on Steam by AznHisoka on Wednesday July 04, 2018 @07:35AM (#17456615)
Isnt Amazon doing a similar thing by showing the Sales rank of a product then?

And Twitter recently now lets you see the ads for any company/competitor as well. its not sales figures but its still competitor intelligence.

Re:Using achievement stats to estimate sales on Steam by dalore on Wednesday July 04, 2018 @08:20AM (#17456807)
In those cases the companies chose to expose that information as it helps sellers and drivers competition amongst the competitors.
Re:Using achievement stats to estimate sales on Steam by AznHisoka on Wednesday July 04, 2018 @08:24AM (#17456825)
How do I then remove my Sales rank from being being displayed? I sell a few products there and I never chose to expose that rank to anyone.
Re:Using achievement stats to estimate sales on Steam by ikeboy on Wednesday July 04, 2018 @08:27AM (#17456845)
Switch to a category that doesn't have ranks, rare but some products don't have it.

If you're one of the top 10k products within a main category Amazon will release it in their public data anyway though.

Re:Using achievement stats to estimate sales on Steam by CamTin on Wednesday July 04, 2018 @01:02PM (#17458270)
They must have reasons for not wanting this figure to be public. If they didn't, they would just expose it directly via the API or in the UI instead of making people solve a fun math puzzle to tease it out of other figures.
Re:Using achievement stats to estimate sales on Steam by derefr on Wednesday July 04, 2018 @02:23PM (#17458712)
If they're the only one with the dataset, they can sell that dataset to publishers. If someone else scrapes the dataset from them, though, then that person or organization can then also sell it to publishers, reducing its market value when Valve goes to sell it.

This is the same reason LinkedIn makes it so hard to scrape their graph.

Re:Using achievement stats to estimate sales on Steam by CamTin on Wednesday July 04, 2018 @02:27PM (#17458731)
Sure, that's one possible reason. Whatever their actual motivations, one thing that we know for sure is that the don't want this data out there. Therefore it should be no surprise that they dull the accuracy on achievements as soon as it is reported that this can be used to divine the numbers. There's not a logically-consistent position Valve could have where they both a) don't want to provide the data directly, but are b) totally fine with you determining it with extreme precision based on other available data. Such a Valve does not and cannot exist.
Re:Using achievement stats to estimate sales on Steam by larsiusprime on Wednesday July 04, 2018 @07:25PM (#17459962)
I know tons of developers and publishers and I have never ever heard of Valve selling their dataset to publishers. Also based on the TOS that I signed with them as a developer (and presumably is the same for every other developer) it would be a violation of their contract with me to share my stats without my permission.
Re:Using achievement stats to estimate sales on Steam by phobosdeimos on Wednesday July 04, 2018 @05:15PM (#17459479)
Publishers and developers don't want sales numbers public because gamers will quickly pick up on low sales and meme you to death. Any bad exposure on the internet can sink a game real fast.
Re:Using achievement stats to estimate sales on Steam by anonnyj on Wednesday July 04, 2018 @09:08AM (#17457046)
Random thought about the accuracy: If you published a game on Steam, you've probably noticed that many owners haven't cleared even "opened up the game" tier achievements.
Re:Using achievement stats to estimate sales on Steam by bartervg on Wednesday July 04, 2018 @10:47PM (#17460551)
Yes, this is an issue.

One example is FreeFly Burning, which according to the achievement estimation method has 739 players [1]. Yet, the game has sold 25,562 keys through Go Go Bundles, and more through other bundles. Go Go Bundles lists the number of bundles delivered [2] and each bundle includes every game (in contrast to Humble Bundle that has tiers). This is consistent with Steamspy's estimate of 20,000 - 50,000 owners [3]. Those bundle buyers likely never downloaded the game and perhaps haven't even activated the game on Steam.

With exceptions like F2P games or free weekend games: Units Sold >= Owners >= Players

[1] https://barter.vg/i/40567/a/

[2] https://www.gogobundle.com/latest/bundles/bundlemega-7

[3] https://steamspy.com/app/683530

Re:Using achievement stats to estimate sales on Steam by larsiusprime on Wednesday July 04, 2018 @03:08PM (#17458925)
I reviewed the algorithm, and as long as you have SOME users at all registering some amount of achievements, the method is quite accurate. Even if most players register no achievements, they still count towards the overall pool of "achievers" and provide a solid base for doing the calculation.

You can verify this yourself with the supplied source code.

The only cases where it can fail are:

1) When there is no achievement data

2) When there are not enough independent achievement %'s.

2 Can happen for games with very few achievements -- but in practice, if a game has even 3 achievements the chances of this happening become very small.

TL;DR -- for very nearly every game on Steam, this was an extremely accurate method for estimating the number of "achievers" for that game. The tricky bit is correlating "achievers" with a metric you actually care about, like e.g. actual sales.

Re:Using achievement stats to estimate sales on Steam by ant6n on Wednesday July 04, 2018 @05:37PM (#17459569)
Isn't that the point of anonnyj? I.e. you don't know which % of sales are made without even registering a single achievement.
Re:Using achievement stats to estimate sales on Steam by larsiusprime on Wednesday July 04, 2018 @07:30PM (#17459975)
Yes you do. Because even if they don't register an achievement, they still show up as part of the total population of possible "achievers". I've verified this from my own stats as a developer and that of other developers.

As long as any portion of the userbase registers some achievement data, you can (well, could) get scarily accurate stats.

The only "inaccurate" part is that the stat you're measuring -- "achievers" -- doesn't exactly correlate to any stat we directly care about, like sales. In all the measuring I did against known private sales data, it was always higher than the # of people who had downloaded the game, and the # of people who had played it for at least 1 minute. My best guess is family sharing inflates the "achievers" stat.

Re:Using achievement stats to estimate sales on Steam by pixl97 on Wednesday July 04, 2018 @09:53AM (#17457251)
I've bought many games on sale and never installed them. Maybe one day I will, maybe I wont. My take is games that never have sales are more likely to be installed and ran.
Re:Using achievement stats to estimate sales on Steam by GuB-42 on Wednesday July 04, 2018 @10:24AM (#17457416)
Also, bundles.

I have a dozen of game I have never played, and will probably never play. They were part of a bundle that was cheaper than the individual game(s) I wanted.

Re:Using achievement stats to estimate sales on Steam by hhmc on Wednesday July 04, 2018 @01:22PM (#17458376)
Also, some games had their achievements added after release (in some cases years). Achievements don't usually get backfilled to my knowledge, so any players with playtime before achievements will be underrepresented.
Re:Using achievement stats to estimate sales on Steam by larsiusprime on Wednesday July 04, 2018 @07:45PM (#17460019)
This isn't the case, actually. Testing it against known sales data, the "achievers" stat derived by this method was ALWAYS higher than the # of ppl who had downloaded the game, or the # of ppl who had played it for at least 1 minute (these stats are known privately to developers).

We don't know exactly how the total "achievers" pool for a game is calculated, but we know it isn't a simple measure of people who had achieved at least one achievement. It's almost certainly some measure of the total population that is able to register achievements, regardless of whether they have registered any.

It probably includes something like family sharing accounts, so it doesn't exactly correspond to actual sales directly, but it seems to be roughly in the middle between the "players" and "owners" stats SteamSpy used to report. Sometimes it would overshoot both, but it was never less than "players."

Re:Using achievement stats to estimate sales on Steam by AdmiralAsshat on Wednesday July 04, 2018 @03:09PM (#17458937)
Yep. I even wonder if Steam has some kind of counter that would show me "Number of games you own that you've never played." I would probably be horrified.
Re:Using achievement stats to estimate sales on Steam by Deimorz on Wednesday July 04, 2018 @03:15PM (#17458963)
SteamDB's calculator can tell you that: https://steamdb.info/calculator/

It splits out a chart showing you how many games you have in a few different playtime categories ("12 or more hours", "6 to 12 hours", "Never played", etc.)

Re:Using achievement stats to estimate sales on Steam by lainga on Wednesday July 04, 2018 @04:18PM (#17459264)
There was an event a few weeks ago where Steam prompted you to play a game in your library that you had never opened or not opened in a long time.
Re:Using achievement stats to estimate sales on Steam by kelukelugames on Wednesday July 04, 2018 @03:18PM (#17458981)
Makes me sad that Valve keeps on trying to kill Steam Spy.
Re:Using achievement stats to estimate sales on Steam by KaoruAoiShiho on Wednesday July 04, 2018 @03:20PM (#17458990)
Valve didn't try to kill Steam Spy... GDPR killed steam spy.
Re:Using achievement stats to estimate sales on Steam by kelukelugames on Wednesday July 04, 2018 @03:41PM (#17459073)
That's not the consensus from game devs at the moment. Perhaps they are wrong.
Re:Using achievement stats to estimate sales on Steam by KaoruAoiShiho on Wednesday July 04, 2018 @04:38PM (#17459331)
Source on consensus?