Also, our goal is not to deceive Google in anyway - a lot of our tests are related to increasing CTR (which is testing humans) and on-page times. (again testing humans) Overall we're trying to make pages better according to Google guidelines -- which leads to a better experience for users.
Some more explanation of how it works here: https://www.rankscience.com/how-it-works
> RankScience sits next to your website, making thousands of experiments to tweak your HTML in order to improve your page rankings.
This seems to me an attempt at reverse engineering the ranking algorithm. Is my interpretation correct? And if so, is this allowed / in scope of the guidelines?
Of course, the final determination is entirely up to Google, but this seems kind of a risky game to play.
From https://support.google.com/webmasters/answer/7238431?hl=en
> Best practices for website testing with Google Search
> The amount of time required for a reliable test will vary depending on factors like your conversion rates, and how much traffic your website gets; a good testing tool should tell you when you’ve gathered enough data to draw a reliable conclusion. Once you’ve concluded the test, you should update your site with the desired content variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup. If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.
Next to this, the advice is to use rel="canonical" to avoid duplicate issues with Googlebot crawling your variations. When using rel="canonical" this should not show you how a variation influences ranking.
> If you’re running an A/B test with multiple URLs, you can use the rel=“canonical” link attribute on all of your alternate URLs to indicate that the original URL is the preferred version. We recommend using rel=“canonical” rather than a noindex meta tag because it more closely matches your intent in this situation.