Re: [alsa-devel] [RFC] ALSA vs. dedicated char device for a USB Audio Class gadget driver
Hi Hal,
first of all, thanks for your answer.
On Thursday 14 May 2009 20:18:07 Hal Murray wrote:
I need an API to transfer audio data from userspace to kernelspace. I initially thought about ALSA, but it turns out some assumptions made by ALSA are not fulfilled by my system. One of the most serious problems is that the UAC gadget driver doesn't have any audio clock. The only hardware clock available is the USB device controller interrupts generated at the USB transfer rate, and those are much faster than the audio sample rate. This will cause buffer underruns that I need to handle.
You don't need a clock. The data will come to you at the right rate. All you need to do is pass it on when you have enough to fill up a buffer. The buffer size is fixed. It's part of the spec for the device you are emulating.
I'm not emulating any device. The buffer size is up to me, and I actually have a fixed number of small buffers, but that shouldn't make a difference.
Assume that you get samples one at a time each time the source clock ticked. The normal state of your system would be to have a buffer that is partially filled. When a new sample fills up the buffer, you would move it from the input side to the ready-for-USB queue and setup a new buffer for future input samples.
Soon the USB side will read the queued buffer, you free that buffer and you are now back to the normal state of collecting input data.
You are in trouble if that doesn't happen before the nest buffer is ready, that is the ready-for-USB queue should normally be empty. If you can't keep it empty it will eventually overflow. Short chunks of time where the queue builds up might be OK. For debugging, you should probably count them.
Now consider the case where you get several samples at a time rather than one. Logically, copy them over one at a time. That turns into a burst of source clocks, but the average will work out right. Your ready-for-USB queue should still be empty most of the time.
If that doesn't make sense, I'll try again. Consider something like audio over a network. There is no audio clock on the network. The receiver can derive the source clock by watching the data stream.
This is more or less what I'm trying to do, with the difference that I don't move samples from the ALSA ring buffer when they arrive but when the USB layer asks for more data.
My trouble is that I don't get notified when new samples are written to the ALSA ring buffer, and I'm not sure to find out how many samples are present in the buffer. If I could get some kind of "sample written" notification, or better, compute the size of data present in the ring buffer, my problem would be (mostly) solved. From what I understand, such a notification isn't possible when the ALSA ring buffer is mmap'ed, as ALSA itself doesn't get notified.
This makes the second approach (reading the number of available samples) better, except that some kind of "free-wheeling" mode makes it impossible under some circumstances. From what Takashi Iwai explained in a mail to Jon Smirl on alsa-devel ("appl_ptr and DMA overrun at end of stream"), "we have also mode when appl_ptr is not updated at all (when stop_threshold == boundary)". This free-wheeling mode seems to be used by dmix.
Best regards,
Laurent Pinchart
On Thu, May 14, 2009 at 1:58 PM, Laurent Pinchart laurent.pinchart@skynet.be wrote:
Hi Hal,
first of all, thanks for your answer.
On Thursday 14 May 2009 20:18:07 Hal Murray wrote:
I need an API to transfer audio data from userspace to kernelspace. I initially thought about ALSA, but it turns out some assumptions made by ALSA are not fulfilled by my system. One of the most serious problems is that the UAC gadget driver doesn't have any audio clock. The only hardware clock available is the USB device controller interrupts generated at the USB transfer rate, and those are much faster than the audio sample rate. This will cause buffer underruns that I need to handle.
You don't need a clock. The data will come to you at the right rate. All you need to do is pass it on when you have enough to fill up a buffer. The buffer size is fixed. It's part of the spec for the device you are emulating.
I'm not emulating any device. The buffer size is up to me, and I actually have a fixed number of small buffers, but that shouldn't make a difference.
Hi Laurent,
The problems might be clearer to you if you do the output side first. You have to define one or more alt/interfaces with isoc OUT sizes to match the sample rate. Some rates are straight forward and don't care much about the buffer/packet sizes like 45,000 HZ. However the 44100 hz rate (and 22050 and 11025) made popular by CDs is interesting because it doesn't divide by 1000, the FS frame rate. So a driver must output 9 packets of 44 samples (times the number of bytes in sample width times stereo/mono) and 1 of 45 samples. Typically USB chips determine the sample rate by the interface and the number of bytes coming per frame.
This is a challenging project, compounded by the audio usb class spec. In my opinion audio is the worst, hardest to understand spec. Second is the HID-PID spec.
Regards, Steve
Hi Steve,
On Thursday 14 May 2009 23:12:41 Steve Calfee wrote:
On Thu, May 14, 2009 at 1:58 PM, Laurent Pinchart
Hi Hal,
first of all, thanks for your answer.
On Thursday 14 May 2009 20:18:07 Hal Murray wrote:
I need an API to transfer audio data from userspace to kernelspace. I initially thought about ALSA, but it turns out some assumptions made by ALSA are not fulfilled by my system. One of the most serious problems is that the UAC gadget driver doesn't have any audio clock. The only hardware clock available is the USB device controller interrupts generated at the USB transfer rate, and those are much faster than the audio sample rate. This will cause buffer underruns that I need to handle.
You don't need a clock. The data will come to you at the right rate. All you need to do is pass it on when you have enough to fill up a buffer. The buffer size is fixed. It's part of the spec for the device you are emulating.
I'm not emulating any device. The buffer size is up to me, and I actually have a fixed number of small buffers, but that shouldn't make a difference.
Hi Laurent,
The problems might be clearer to you if you do the output side first. You have to define one or more alt/interfaces with isoc OUT sizes to match the sample rate. Some rates are straight forward and don't care much about the buffer/packet sizes like 45,000 HZ. However the 44100 hz rate (and 22050 and 11025) made popular by CDs is interesting because it doesn't divide by 1000, the FS frame rate. So a driver must output 9 packets of 44 samples (times the number of bytes in sample width times stereo/mono) and 1 of 45 samples. Typically USB chips determine the sample rate by the interface and the number of bytes coming per frame.
The USB side is already implemented. I need an isochronous endpoint with a packet size just a bit bigger than what would be required by the sample rate. The amount of data per packet will vary (some might even be empty), but that won't be a problem.
I now need to fill the packets with data. When a packet has been successfully transferred on USB the USB device controller will notify me with an interrupt. I then have to copy audio data from the ALSA ring buffer to the USB request buffer and queue the request for transmission (a USB request, in the USB gadget device driver terminology, is a data structure that describes a USB transfer).
To copy data from the ALSA ring buffer I need to know how much data is available at that particular moment, in order to avoid buffer underruns. I'm still not sure how to do that.
This is a challenging project, compounded by the audio usb class spec. In my opinion audio is the worst, hardest to understand spec. Second is the HID-PID spec.
Challenges are usually interesting :-) With a little luck part of the complexity will be handle outside the UAC gadget driver, either on the host side or by the userspace application on the device side.
Best regards,
Laurent Pinchart
Laurent Pinchart wrote:
To copy data from the ALSA ring buffer I need to know how much data is available at that particular moment, in order to avoid buffer underruns.
This is not how ALSA drivers are supposed to work. The framework was designed for DMA devices that run asynchronously and just take data out of the ring buffer whenever they need them. Your driver should work in the same way, i.e., you queue a packet of data when you need to queue it.
Best regards, Clemens
Hi Clemens,
thanks for your answer.
On Friday 15 May 2009 09:01:48 Clemens Ladisch wrote:
Laurent Pinchart wrote:
To copy data from the ALSA ring buffer I need to know how much data is available at that particular moment, in order to avoid buffer underruns.
This is not how ALSA drivers are supposed to work. The framework was designed for DMA devices that run asynchronously and just take data out of the ring buffer whenever they need them. Your driver should work in the same way, i.e., you queue a packet of data when you need to queue it.
I know that ALSA is designed for asynchronous operation. This is fine for devices that provide an audio clock (all sound cards do), but leads to a few issues for devices that don't, like this one.
When streaming audio data from a file to USB, there is simply no audio clock in the system. When streaming audio data from a real audio device to USB, the audio clock comes from the real audio device and is beyond control of the USB Audio Class gadget driver.
I can see two ways around this.
1. Use an asynchronous isochronous endpoint to stream data. The USB Audio Class gadget driver will either have to be provided with a clock from userspace, or stream data as fast as it can, needing a way to check the amount of data present in the buffer (as explained in my previous e-mail).
2. Use a synchronous isochronous endpoint to stream data. The USB Audio Class gadget driver will derive its clock from the USB Start Of Frame clock and the sample rate, and send a fixed number of audio samples per packet. In this case the userspace application will have to resample audio data to match the source and sink rates. The issue with this solution is that I'm not sure if the USB stack provides the necessary information. I need to fill USB packets with audio data and queue them for transmission, but I don't know when they will be transmitted, making it difficult to adjust the amount of data per packet based on the sample rate.
As a side note, I'm not sure how difficult audio rate matching is, I suppose the userspace ALSA API provides enough information to match rates, but I'd appreciate if someone could confirm this (and possibly provide a few pointers to related documentation).
Best regards,
Laurent Pinchart
Laurent Pinchart wrote:
I know that ALSA is designed for asynchronous operation. This is fine for devices that provide an audio clock (all sound cards do), but leads to a few issues for devices that don't, like this one.
When streaming audio data from a file to USB, there is simply no audio clock in the system.
In this case, you can use the USB frame clock as audio clock, i.e., whenever one frame has been completed, you also submit audio data for one USB frame (one millisecond).
This would be a synchronous endpoint.
When streaming audio data from a real audio device to USB, the audio clock comes from the real audio device and is beyond control of the USB Audio Class gadget driver.
In this case, you have _two_ clocks, the clock of the actual audio device, and the USB SOF clock. What you need to know is the relative speed of these clocks, because that determines the number of samples per USB frame.
I can see two ways around this.
- Use an asynchronous isochronous endpoint to stream data. The USB Audio
Class gadget driver will either have to be provided with a clock from userspace, or stream data as fast as it can,
The latter is not allowed by the USB specification; the data rate must change only gradually, i.e., the difference in successive frames must be no more than +/- one sample.
needing a way to check the amount of data present in the buffer (as explained in my previous e-mail).
The _current_ amount of valid data does not help because applications can write data irregularly. You do not want to measure how fast the application writes data into your buffer, but the actual speed of the actual audio device.
I'd guess that your gadget driver does not have access to the real audio device, so it is the application that must do the measurement.
The ALSA API has no mechanism to tell the driver how fast to play, so it looks as if you have to write you own device driver.
Best regards, Clemens
Hi Clemens,
On Friday 15 May 2009 17:07:30 Clemens Ladisch wrote:
Laurent Pinchart wrote:
I know that ALSA is designed for asynchronous operation. This is fine for devices that provide an audio clock (all sound cards do), but leads to a few issues for devices that don't, like this one.
When streaming audio data from a file to USB, there is simply no audio clock in the system.
In this case, you can use the USB frame clock as audio clock, i.e., whenever one frame has been completed, you also submit audio data for one USB frame (one millisecond).
This would be a synchronous endpoint.
Using a synchronous endpoint in this case sounds easy. I'll probably do that.
When streaming audio data from a real audio device to USB, the audio clock comes from the real audio device and is beyond control of the USB Audio Class gadget driver.
In this case, you have _two_ clocks, the clock of the actual audio device, and the USB SOF clock. What you need to know is the relative speed of these clocks, because that determines the number of samples per USB frame.
If I use a synchronous endpoint, isn't the number of samples per frame determined by the nominal sampling rate and the nominal SOF frequency ? With the SOF clock running at 1kHz, I expect a synchronous endpoint for a 16 bits mono 48kHz stream to deliver exactly 48 frames (96 bytes) per USB frame.
I can see two ways around this.
- Use an asynchronous isochronous endpoint to stream data. The USB Audio
Class gadget driver will either have to be provided with a clock from userspace, or stream data as fast as it can,
The latter is not allowed by the USB specification; the data rate must change only gradually, i.e., the difference in successive frames must be no more than +/- one sample.
Thanks for pointing this out.
needing a way to check the amount of data present in the buffer (as explained in my previous e-mail).
The _current_ amount of valid data does not help because applications can write data irregularly. You do not want to measure how fast the application writes data into your buffer, but the actual speed of the actual audio device.
I'd guess that your gadget driver does not have access to the real audio device, so it is the application that must do the measurement.
That's right. The gadget driver doesn't care if data comes from a file or a real audio device.
The ALSA API has no mechanism to tell the driver how fast to play, so it looks as if you have to write you own device driver.
If I use a synchronous endpoint I can solve the problem on the driver side. There would still be a problem on the application side, as the audio source clock will obviously not be synchronous with the USB SOF clock. The userspace application will request audio data at a nominal rate equal to the nominal sample frequency advertised on USB, and will need to perform rate matching between its source (real audio device) and sink (USB driver). Is that (at least relatively) easy to do with ALSA ? I suppose this is a common use case for all audio streaming applications such as internet radios, as in that case audio data streamed over the network will not be synchronous with the sound card audio clock and thus need to be resampled in the audio player before being sent to the sound card.
Best regards,
Laurent Pinchart
Laurent Pinchart wrote:
On Friday 15 May 2009 17:07:30 Clemens Ladisch wrote:
Laurent Pinchart wrote:
When streaming audio data from a real audio device to USB, the audio clock comes from the real audio device and is beyond control of the USB Audio Class gadget driver.
In this case, you have _two_ clocks, the clock of the actual audio device, and the USB SOF clock. What you need to know is the relative speed of these clocks, because that determines the number of samples per USB frame.
If I use a synchronous endpoint, isn't the number of samples per frame determined by the nominal sampling rate and the nominal SOF frequency ?
Yes, but in _this_ case, the second case, you cannot use a synchronous endpoint. Since the real audio device controls the rate, this would be an asynchronous endpoint.
I'd guess that your gadget driver does not have access to the real audio device, so it is the application that must do the measurement.
That's right. The gadget driver doesn't care if data comes from a file or a real audio device.
I'd guess you don't want to restrict your gadget to reading from a file, so you'll have to implement an asynchronous endpoint, and reading from a file is just an uninteresting special case.
The ALSA API has no mechanism to tell the driver how fast to play, so it looks as if you have to write you own device driver.
If I use a synchronous endpoint I can solve the problem on the driver side. There would still be a problem on the application side, as the audio source clock will obviously not be synchronous with the USB SOF clock. The userspace application will request audio data at a nominal rate equal to the nominal sample frequency advertised on USB, and will need to perform rate matching between its source (real audio device) and sink (USB driver). Is that (at least relatively) easy to do with ALSA ?
It is possible, but not trivially easy.
To measure the rate of the real audio device, you have to continually (or regularly) correlate the sample position (derived from snd_pcm_delay()) and some time stamp. Using the computer's time would introduce a third clock, so I'd guess you should use the USB SOF interrupts as timer ticks (so your driver should notify the application of SOFs).
With these measurements, you have the audio device rate, measured in (i.e., relative to) the USB SOF rate. With this relation, you can do the rate matching; or you could use it to adjust the packet sizes of the asynchronous endpoint, which I think is much easier.
I suppose this is a common use case for all audio streaming applications such as internet radios, as in that case audio data streamed over the network will not be synchronous with the sound card audio clock and thus need to be resampled in the audio player before being sent to the sound card.
Or the player uses a buffer that is big enough so that there is enough data even if the sound card runs faster than the server. :-)
Best regards, Clemens
On Friday 15 May 2009 19:04:10 Clemens Ladisch wrote:
Laurent Pinchart wrote:
On Friday 15 May 2009 17:07:30 Clemens Ladisch wrote:
Laurent Pinchart wrote:
When streaming audio data from a real audio device to USB, the audio clock comes from the real audio device and is beyond control of the USB Audio Class gadget driver.
In this case, you have _two_ clocks, the clock of the actual audio device, and the USB SOF clock. What you need to know is the relative speed of these clocks, because that determines the number of samples per USB frame.
If I use a synchronous endpoint, isn't the number of samples per frame determined by the nominal sampling rate and the nominal SOF frequency ?
Yes, but in _this_ case, the second case, you cannot use a synchronous endpoint. Since the real audio device controls the rate, this would be an asynchronous endpoint.
Even if I resample the audio stream using the SOF clock ? As you pointed before, ALSA was designed around an asynchronous model. I need an audio clock to control the output of audio data. If I can't use the SOF clock, and as my driver can't access the audio input clock, the driver will not be able to copy data asynchronously from the ALSA ring buffer.
I'd guess that your gadget driver does not have access to the real audio device, so it is the application that must do the measurement.
That's right. The gadget driver doesn't care if data comes from a file or a real audio device.
I'd guess you don't want to restrict your gadget to reading from a file, so you'll have to implement an asynchronous endpoint, and reading from a file is just an uninteresting special case.
Then I need to either provide the driver with an audio clock from the userspace application (ALSA doesn't have any mechanism for that, as it assumes the device provides an audio clock), or check periodically how much data is available in the ALSA ring buffer. This is made even more complex by the fact that packets can vary by +/- 1 byte only according to your last e-mail.
The ALSA API has no mechanism to tell the driver how fast to play, so it looks as if you have to write you own device driver.
If I use a synchronous endpoint I can solve the problem on the driver side. There would still be a problem on the application side, as the audio source clock will obviously not be synchronous with the USB SOF clock. The userspace application will request audio data at a nominal rate equal to the nominal sample frequency advertised on USB, and will need to perform rate matching between its source (real audio device) and sink (USB driver). Is that (at least relatively) easy to do with ALSA ?
It is possible, but not trivially easy.
To measure the rate of the real audio device, you have to continually (or regularly) correlate the sample position (derived from snd_pcm_delay()) and some time stamp. Using the computer's time would introduce a third clock, so I'd guess you should use the USB SOF interrupts as timer ticks (so your driver should notify the application of SOFs).
With these measurements, you have the audio device rate, measured in (i.e., relative to) the USB SOF rate. With this relation, you can do the rate matching; or you could use it to adjust the packet sizes of the asynchronous endpoint, which I think is much easier.
How can I adjust the asynchronous endpoint packet sizes in the driver if I can't access the audio clock there ?
I suppose this is a common use case for all audio streaming applications such as internet radios, as in that case audio data streamed over the network will not be synchronous with the sound card audio clock and thus need to be resampled in the audio player before being sent to the sound card.
Or the player uses a buffer that is big enough so that there is enough data even if the sound card runs faster than the server. :-)
Lazy players :-)
Best regards,
Laurent Pinchart
participants (3)
-
Clemens Ladisch -
Laurent Pinchart -
Steve Calfee