Living with a best-selling Indian phone for 10 days

Re:Living with a best-selling Indian phone for 10 days by bm1362 on Friday July 13, 2018 @08:22PM (#17527947)

I work in mobile performance and have built out an automated testing workflow for our CI system to try and prevent big breaking changes to these types of phones.

The issue with testing these phones is the device introduces lots of variance due to the low hardware standards. We typically test for relative changes in key metrics and want to move to testing different types of phones for usability. This is generally hard because the noise makes the test inconclusive.

[ Parent | on HN ]

Re:Living with a best-selling Indian phone for 10 days by toredash on Saturday July 14, 2018 @01:08AM (#17528933)

How do you this testing? Any public service?

[ Parent | on HN ]

Re:Living with a best-selling Indian phone for 10 days by bm1362 on Saturday July 14, 2018 @08:45PM (#17533175)

We have a lab environment with multiple phones plugged into a single controller. The controller runs a set of Jenkins agents which advertises the different devices as targets for Jenkins jobs.

Plus, lots of tweaking and perf related fixes to the phone. If you're looking for an off-the-shelf solution, check out Headspin or the Google Cloud offerings.

[ Parent | on HN ]

Re:Living with a best-selling Indian phone for 10 days by toredash on Sunday July 15, 2018 @04:57AM (#17534438)

Thanks for the feedback. I'll look into headspin. I'm not into mobile performance or testing so not sure how I would automate and log the performance on different devices. Honestly I'm not sure what tools I would use on the device to run the tests.

[ Parent | on HN ]

Re:Living with a best-selling Indian phone for 10 days by bm1362 on Sunday July 15, 2018 @05:20AM (#17534493)

We are trying to build something similar to CT-scan; except it’s a 1-man team so it’s been a longer journey:

https://code.fb.com/developer-tools/mobile-performance-tooli...

We run functional UI tests with a performance tracer wired up- we already had them setup by feature teams so it was easy to get started. Instead of a Zookeeper scheduler, it’s just Jenkins.

Right now we’ve been able to catch 2-3% relative regressions in things like startup, opening a Google Maps view etc which is pretty good. It was easy to setup and then hard to optimize the variance down so we could get a strong signal.

We’re hiring if any of this is interesting, it’s Golang/San Francisco/Full-Time.

[ Parent | on HN ]