Hi Dan,

On 25.11.2012 04:33, Daniel Griscom wrote:

At 10:32 PM +0100 11/22/12, Daniel Mack wrote:

...

On 21.11.2012 21:13, Daniel Griscom wrote:

...

On to the next problem we're having with our USB audio card. Recap: simple USB Audio Class embedded stereo audio card, running at 44.1kHz or 48kHz, using the 3.6.6 kernel and its associated ALSA modules to drive it, running on Jetway ND9D-2700, with an Atom D2700 and NM10 chipset. Output of "sudo lsusb -v" for card is appended below.

Problem: we're experiencing strange hiccups in the delivery of audio from the motherboard to the card. At 48kHz, the motherboard should send a 6-frame packet to the card once per high-speed microframe (every 125us). This usually works, but sometimes there are two closely related failures:

1. Once in a while (every few seconds or minutes) a single

packet will be missed

2. Once in a longer while (every few minutes or hours), the

system will go into a continuing error condition where three of every eight packets are missed.

In both cases, "missed" means that ALSA consumes the output audio from my motherboard application, but never sends a packet to the audio card. No USB errors, no logged messages; it just disappears.

I missed that detail before. What makes you so sure ALSA actually consumes audio it never sends out? Are you sending a test pattern which is interrupted in your USB analyzer traces?

Great idea, and it took me a few days to be able to do it (too much turkey to be dealt with). The answer is yes: ALSA is consuming data that doesn't make it onto the USB bus.

My app is completely clocked by ALSA, sending out whatever the ALSA output stream will accept, and taking in whatever the ALSA input stream will offer. I set my app to output stereo triangle waves (in different frequencies), captured the data with the Beagle 1200, and then graphed it as a function of sample count. When things were running fine, the graph looked nice and clean. When the packets were being lost, both lines in the graph periodically jump, skipping values. The sizes of the jumps alternate, matching the pattern I cited where in each set of eight microframes two adjacent frames are skipped, and then a single frame is skipped.

If ALSA hadn't taken the data, my app would just wait until it did, with no data lost (although the output graph would still be bad as a function of time). So, ALSA (or something deeper) must be discarding the data.

Please do another test and change prepare_outbound_urb() so that instead of calling ep->prepare_data_urb() to fill the URB, fill it directly with some sort of easily recognizable test pattern (you can take the code from "silence" case to access the buffers). Some kind of 16-bit counter should do. And then check the payload with the Beagle and see whether the pattern has gaps.

This test will tell us whether data is in fact lost before it hits the usb audio driver, or if it's dropped by the USB HCD.

Daniel

At 5:39 PM +0100 11/25/12, Clemens Ladisch wrote:

Daniel Mack wrote:

...

...

On 21.11.2012 21:13, Daniel Griscom wrote:

...

1. Once in a while (every few seconds or minutes) a single

packet will be missed

2. Once in a longer while (every few minutes or hours), the

system will go into a continuing error condition where three of every eight packets are missed.

[...] This test will tell us whether data is in fact lost before it hits the usb audio driver, or if it's dropped by the USB HCD.

These look like hardware errors.

It's possible. This is a Jetway NF9D-2700 motherboard, with USB handled by an NM10, and there are other Linux issues with the hardware (e.g. the PowerVR GPU isn't supported in recent Ubuntu versions). But, the test results in my previous email (sent after yours) imply that it's a software issue.

You might want to try to check the frames' status in retire_playback_urb(),

I've looked that function over, but it's not clear where in the various structures I'd find the status. Suggestion?

but this will show only DMA/ FIFO problems on the host; the HC will not be able to detect whether the receiving device has handled the packet.

You mean, if the data is put on the USB bus, but the receiving device (my audio card) loses it, there's no way the above will show this? The missing data never appears on the USB bus, so I'm pretty sure that isn't the problem here.

By the way, I'd stated that the pattern of missing packets repeats every eight microframes, and never varies. I've found that that isn't completely true; it sometimes does vary, but not by much. By microframe numbers within a frame, here are the two patterns I've seen:

0 - Miss 1 - Miss 2 - Send 3 - Send 4 - Send 5 - Miss 6 - Send 7 - Send

0 - Miss 1 - Send 2 - Send 3 - Send 4 - Send 5 - Miss 6 - Miss 7 - Send

(I've spent a while trying to concoct some sort of rule for these patterns based on the microframe numbers' bits, with no success. Oh, well.)

As always, thanks for your continuing help, Dan

At 1:43 PM +0100 11/25/12, Daniel Mack wrote:

Please do another test and change prepare_outbound_urb() so that instead of calling ep->prepare_data_urb() to fill the URB, fill it directly with some sort of easily recognizable test pattern (you can take the code from "silence" case to access the buffers). Some kind of 16-bit counter should do. And then check the payload with the Beagle and see whether the pattern has gaps.

This test will tell us whether data is in fact lost before it hits the usb audio driver, or if it's dropped by the USB HCD.

I did the test, and expected to either a) see the incrementing data be contiguous even though the USB packet transmission was stuttering, or b) see the incrementing data have gaps where the "missing" packets were. However, what I saw was c): the USB packets never misbehaved.

Here's the code I used in /usb/endpoint.c, prepare_outbound_urb():

static short left = 0; static short right = 0x0404; short *sp;

...

if (ep->prepare_data_urb) {

#if 1 ep->prepare_data_urb(ep->data_subs, urb); #else /* Fake constantly increasing samples */ unsigned int offs = 0; for (i = 0; i < ctx->packets; ++i) { int counts;

      if (ctx->packet_size[i])
	counts = ctx->packet_size[i];
      else
	counts = snd_usb_endpoint_next_packet_size(ep);

      urb->iso_frame_desc[i].offset = offs * ep->stride;
      urb->iso_frame_desc[i].length = counts * ep->stride;
      offs += counts;
  }

  urb->number_of_packets = ctx->packets;
  urb->transfer_buffer_length = offs * ep->stride;
  for (i = 0, sp = (short *)(urb->transfer_buffer);
       i < offs * ep->stride; /* no increment */) {
    *sp++ = left++;
    i += sizeof(short);
    *sp++ = right++;
    i += sizeof(short);
  }

#endif } else {

With the "#if 1", the code is stock, and about 30% of my tests resulted in the stuttering USB packets (but with data from my application). With it changed to "#if 0", I generated and saw the incrementing data, but the USB packets never misbehaved (even after more than 20 tries). I switched back and forth, and the problem clearly appeared and disappeared.

I think I should give more information about my tests. This problem is very intermittent, and difficult to reproduce. However, I found a case where it often happens: at boot, where my app starts up right about when the USB devices are starting up. So, each of the "tests" above consisted of me rebooting my system and watching the Beagle's trace for good versus bad packet patterns.

Thanks again for the continuing help, Dan

At 8:19 AM +0100 11/26/12, Daniel Mack wrote:

On 25.11.2012 23:23, Daniel Griscom wrote:

...

At 1:43 PM +0100 11/25/12, Daniel Mack wrote:

...

Please do another test and change prepare_outbound_urb() so that instead of calling ep->prepare_data_urb() to fill the URB, fill it directly with some sort of easily recognizable test pattern (you can take the code from "silence" case to access the buffers). Some kind of 16-bit counter should do. And then check the payload with the Beagle and see whether the pattern has gaps.

This test will tell us whether data is in fact lost before it hits the usb audio driver, or if it's dropped by the USB HCD.

I did the test, and expected to either a) see the incrementing data be contiguous even though the USB packet transmission was stuttering, or b) see the incrementing data have gaps where the "missing" packets were. However, what I saw was c): the USB packets never misbehaved.

Note that this is a potential difference between your two cases.

prepare_playback_urb() from pcm.c will not always set urb->number_of_packets to the maximum value but cut packets short at PCM period boundaries.

Could you move your pattern generating code down to that function? Just look for the memcpy() calls there.

OK, this time I got some real results. I can still get the USB packet stream to misbehave, and indeed the synthesized data is being lost. The output data present on the USB bus smoothly increments during each packet, continuing smoothly in adjacent packets. But, when one or two packets are missed, the next output data is numbered as if the packets (6 stereo frames each) had been generated but lost.

Here's my code in linux-3.6.6/sound/usb/pcm.c, prepare_playback_urb():

static short left = 0; static short right = 0x0404; short *sp;

...

bytes = frames * stride; #define FAKE_DATA #ifdef FAKE_DATA for (i = 0, sp = (short *)(urb->transfer_buffer); i < bytes; /* no increment */) { *sp++ = left++; i += sizeof(*sp); *sp++ = right++; i += sizeof(*sp); } #else if (subs->hwptr_done + bytes > runtime->buffer_size * stride) { /* err, the transferred area goes over buffer boundary. */ unsigned int bytes1 = runtime->buffer_size * stride - subs->hwptr_done; memcpy(urb->transfer_buffer, runtime->dma_area + subs->hwptr_done, bytes1); memcpy(urb->transfer_buffer + bytes1, runtime->dma_area, bytes - bytes1); } else { memcpy(urb->transfer_buffer, runtime->dma_area + subs->hwptr_done, bytes); } #endif subs->hwptr_done += bytes;

Did I place the generation code in the correct location?

And, what should I look at next?

Thanks again, Dan

At 12:48 PM +0100 11/27/12, Daniel Mack wrote:

As Clemens said, check the error fields in the isochronous frames when the urb is given back to the driver. For isochronous data, there's nothing the driver can do about failed transmissions, hence there's no code to handle such conditions. But it might help understand what's going on.

Well, I inserted code into retire_playback_urb() as Clemens suggested a few days ago:

int est_delay, i;

/* Check status of submitted urbs */ for (i = 0; i < urb->number_of_packets; i++) { if (urb->iso_frame_desc[i].status) { snd_printk(KERN_ERR "packet %d failed: %d\n", i, urb->iso_frame_desc[i].status); } }

/* ignore the delay accounting when procssed=0 is given, i.e.

And, hit gold (lead?):

[ 432.591698] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.591702] ALSA sound/usb/pcm.c:1163 packet 1 failed: -18 [ 432.592684] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.592702] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.592706] ALSA sound/usb/pcm.c:1163 packet 1 failed: -18 [ 432.593684] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.593701] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.593705] ALSA sound/usb/pcm.c:1163 packet 1 failed: -18 [ 432.594680] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18

Error 18 is EXDEV: "Cross-device link". From an alsa-user post by Clemens a couple of years ago:

This error code is documented in Documentation/usb/error-codes.txt: | -EXDEV ISO transfer only partially completed | look at individual frame status for details

Well, this _is_ the individual frame status.

Anyway, a look at the source code shows that several HCDs use this code for frames that were not handled by the controller, probably because they were submitted too late.

Which controller are you using?

Is there anything that could cause the controller to skip frames, such as high traffic on the (PCI) bus?

The answer to that final question: I don't know, but I've been concerned about PCI traffic for a while, and I don't know how I'll address it.

So, is it clear that this is a USB (software or hardware) problem? Should I move this discussion to the linux-usb list?

Thanks for all the help, Dan

At 6:21 PM +0100 11/27/12, Clemens Ladisch wrote:

Daniel Griscom wrote:

...

...

[ 432.591698] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.591702] ALSA sound/usb/pcm.c:1163 packet 1 failed: -18 [ 432.592684] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.592702] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.592706] ALSA sound/usb/pcm.c:1163 packet 1 failed: -18 [ 432.593684] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.593701] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18 [ 432.593705] ALSA sound/usb/pcm.c:1163 packet 1 failed: -18 [ 432.594680] ALSA sound/usb/pcm.c:1163 packet 0 failed: -18

These packet numbers never get above 1. Are you using two microframes per URB?

We are using a bInterval value of 1 (on a high-speed device). We've recently tried bumping this up to 2, and the problem was no longer evident, but given the difficulty in reproducing it I wasn't at all clear that we hadn't just moved it to a different set of reproduce conditions.

... is that what you meant?

What is the ALSA buffer size?

We're handing a value of 3072 to snd_pcm_hw_params_set_buffer_size_near().

Regards, Clemens

Thanks, Dan

At 8:46 PM +0100 11/27/12, Clemens Ladisch wrote:

Daniel Griscom wrote:

...

We are using a bInterval value of 1 (on a high-speed device). We've recently tried bumping this up to 2, and the problem was no longer evident

This might indicate that the controller has problems reading the packet data from memory; there's a considerable overhead for each memory transaction, and the controller needs to update the DMA lists. (The pattern might be caused by some data, which arrives too late, interfering with the reading of the next packet.)

So, might this whole thing have been caused by setting bInterval to 1 in the device descriptor?

If 2 is the first save interval, consider using 4.

Interval lengths are 2 ^ (bInterval - 1) microframes, so we went from 1 microframe to 2. Should I go right to a bInterval of 2 (4 microframes)?

For that matter, full-speed devices work fine with one packet per frame. There isn't a problem with buffering one millisecond of data in the device, is it?

We are indeed trying to squeeze the milliseconds out of the audio chain, but getting it robust is obviously the priority.

(The only reason to use smaller intervals is if you really require sub- millisecond latencies, or if the bigger packets would eat up too much bandwidth in one of the microframes. The cost of smaller intervals is more overhead on the host, on the bus, and in the device.)

We have plenty of bus bandwidth, and I'd thought we had plenty of CPU power, but perhaps not.

Thanks, Dan