I'm the author. My testing methodology was to run the round-trip latency test using OboeTester: https://github.com/google/oboe/tree/master/apps/OboeTester/d....

It's a simple test: produce a tone (either on built-in speaker or over wired headphones if using a loopback dongle), measure the time it takes for that tone to reach the audio input.

For devices which I didn't have access to (our team has a limited number of test devices) I used the figures from Superpowered, but with some assumptions/rules:

#1 If AAudio was available on the device I used the measurements from that rather than OpenSL ES #2 No measurements from custom ROMs #3 I used the measurements from the latest version of Android which the OEM had released for that device. For example, if a device was originally released with Lollipop but it was possible to upgrade to Marshmallow then I used the figures for Marshmallow.

For touch screen latency measurements I used a WALT device (https://github.com/google/walt)

I'm actually very interested in the details. I have a few questions, in case you have cycles to help me understand.

# audio latency

From your comment and the original "An update on Android's audio latency" article, there seems to be 2 different ways to calculate "audio latency".

* 1. play a well-known sound (a tone), listen for it in real-time

IIUC, the exact operation of this would be something like this:

  open recorder
  loop:
    timestamp1 <- getclock()
    play audio file 
    keep listening to input frames until detecting audio file signature
    timestamp2 <- getclock()
    sample := timestamp2 - timestamp1

* 2. measure audio loopback feedback (larsen effect)

Operation should be as follows:

  open recorder
  play audio file
  loop:
    keep listening to input frames for 1-2 seconds
  # analyze captured file
  look for 2x consecutive appearances of the original audio file
  sample := timestamp2 - timestamp1

Q1: Did I get this right? Any hunch about the pros&cons of both approaches?

# downlink/uplink latency

Many of the latency measurements break the latency down between audio downlink and uplink.

Q2: How is this measured?

The android doc discusses using a GPIO to have a zero-latency signal. You swap the signal on a well-known GPIO, and then play a well-known audio file. You then connect the GPIO to a speaker, and measure the latency between the GPIO-fed speaker, and the actual device speaker. As an optimization, you plug both the GPIO and the audio jack that feeds the actual device speaker into an oscilloscope, and get the distance there. Is this how audio downlink latency is measured?

Q3: If this is how downlink latency is measured, how do you implement it in prod devices (no access to the board)? Is this something that can be simulated using the usb-c plug?

# tools

The android doc/some googling points to several tools. I found oboetester, splatency, drrickorang, and google walt. I tested all but the walt one.

The first interesting thing is that oboetest, drrickorang, and walt seem to need an external jack-based dongle.

Q3: Why is this needed for? The audio latency approaches mentioned above should only need a speaker and a mic.

The experience has not been great for any of the tools. Operationally, I'd like to script these tests. Instead, I need to click on GUIs. Also, all the tools I tested seem to fail often, so I have to repeat the experiments multiple times. Also, I miss better documentation on exactly what they do.

Thanks for any help

edit: s/markdown/formatdoc/g

Below is a collaborative answer from myself and my colleague, Phil Burk who is a SWE on the audio framework team.

Thanks for your interest in the details of Android Latency measurement techniques.

Some of the measurements were collected by third parties. We cannot describe their techniques but they are probably similar to ours.

We use OboeTester to measure latency. You can find a description of how to measure Tap-to-Tone Latency and Round Trip Latency in this doc: https://github.com/google/oboe/blob/master/apps/OboeTester/d...

We do not use the Larsen Effect any more because it was too sensitive to variations in gain. We now use a random encoded bit stream that sounds like a short noise burst. We can get a better correlation peak with that signal.

> Many of the latency measurements break the latency down between > audio downlink and uplink.

It is very hard to separate the input and output latency without special hardware (like the WALT device). You can measure combined input+output latency using a loopback test. Input latency tends to be much lower than output latency. This is because, when the full duplex stream is stable, the input buffer is close to empty and the output buffer is close to full. Then, if there is a preemption, the input buffer fills up and the output buffer drains, providing glitch protection.

You can measure touch+output latency using tap-to-tone. The screen touch latency is about 15-30 msec. If you use a hardware MIDI controller (like a keyboard or drum pad) instead of tapping the touch screen then you can get a “touch” latency of about 1 msec (MIDI is a lightweight and low latency protocol). Then you can get a better estimate of just the output latency.

> The android doc discusses using a GPIO to have a zero-latency signal.

We don’t normally use that technique because it requires special hardware.

> I found oboetester, splatency, drrickorang, and google walt.

Our group supports OboeTester.

> The first interesting thing is that oboetest, drrickorang, and walt seem to need an external jack-based dongle.

The lowest latency path is usually over the headphone jack (either through 3.5mm or USB dongle). To test this path you do need a "jack-based dongle" aka loopback adapter.

The reason why wired headphones usually give you lower latency is that OEMs often introduce digital signal processing for the speaker to improve the acoustics/quality, which can introduce additional latency. Side note: this is why you will often see "best with headphones" on games and music apps.

That said, OboeTester will work over the speakers and mic in a quiet room. That is because the new random bit technique is more robust.

> I'd like to script these tests.

OboeTester can be scripted. We use it for continuous integration testing. https://github.com/google/oboe/blob/master/apps/OboeTester/d...

Thanks Don and Phil for the great answer. Some further comments:

* I find interesting that you are getting better correlations when you add a well-known noise burst instead of a less chaotic signal (e.g. a tone or a chirp). In retrospect, it makes sense

* for the uplink/downlink breakup, I see in the Usage.md file in oboetester that you're isolating the downlink measurement with the tap-to-tone experiment. For this experiment, the doc suggests to use (a) the jack to avoid the speaker processing extra latency, and (b) a USB-MIDI input device to replace the touch screen latency (15-30 ms).

2 questions here:

* Q1: I assume that, if instead of the jack, you use a usb-c audio adapter accessory mode, there should be no extra latency either, right? (I'm using late pixel phones)

* Q2: which device are you using for the USB-MIDI input?

Thanks again!

edit: s/markdown/formatdoc/g

> * Q1: I assume that, if instead of the jack, you use a usb-c audio adapter accessory mode, there should be no extra latency either, right? (I'm using late pixel phones)

That's correct, by using either the 3.5mm jack, or USB-C adapter you won't incur any additional latency introduced by DSP to improve the speaker acoustics.

However, the USB path typically does have a few ms higher latency than the 3.5mm jack path.

> * Q2: which device are you using for the USB-MIDI input?

We've used a variety of devices and found the latency differences to be negligible. At the moment I test with an old AKAI LPK25.

> I read through the whole article and that's the most detail given about any testing methodology; I don't see any more description of how these numbers are being measured.

The details usually aren’t that interesting. What you can do is record an input signal, pass the signal through to the output, and wire the output back in to another input.

You end up with two copies of the input signal, one delayed by the total round-trip time relative to the other. It’s fairly easy.

It's certainly interesting because they can have a huge effect on the measurement results. For example, with HDA and I suspect other audio codec systems, what you described can be done with essentially 0 delay by configuring the codec to do the in/out mix itself. If you are measuring output from application buffers to input to application buffers, then how big they are and at what sample rates and bit depths will also affect the latency you see, etc.

Certainly nobody would then agree this is measuring the latency of the audio _system_, it would only measure the latency of the audio device, right?

I think we can give the benefit of the doubt and assume they are not trying to actively deceive.