[alsa-devel] LSB inclusion of ALSA
Hi,
Currently ALSA is a trial use module for LSB (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it is the intention to make ALSA mandatory with the LSB 4.1 release targeted for Q1 2010. One of the requirements of a mandatory module is the existence of a test suite for the module that covers a good chunk of the interfaces provided by the LSB spec. An initial exploratory look at the existing test cases showed that tests appear to be tied to hardware drivers and are interactive.
Is this initial impression correct?
_From an LSB testing perspective, interface testing is very important; i.e. I call the interface with the arguments required and I get the expected result back. Further the tests need to run in an automated fashion and produce a certain output (automated processing). The output can be created by retrofitting existing tests or by specifically writing tests within the LSB test framework.
IMHO it would be best to have tests that are useful to both, the project and the LSB and in a best case scenario we wouldn't have to start over as the existing tests already cover a good part of the interface.
With this as background there are a few questions that have been raised while discussing this during LSB workgroup sessions.
1.) Is there interest from the community side to participate in this effort and accept patches? 2.) Is it reasonable to expect that the existing tests can be used in some way by using a dummy sound device or a sound loop device to verify output? One requirement for LSB testing is that a test has only two states, pass or fail, and a recovery or try again mode is not supported. Further a test must pass or fail. 3.) Would the community be more comfortable if this is an effort that creates new tests separate from the existing HW focused tests? 4.) Is anyone interested in helping with this effort, writing tests, answering questions about ALSA for those not familiar with the interface and or sound in general?
Comments, thoughts, etc. are much appreciated.
Thanks, Robert
Robert Schweikert wrote:
Currently ALSA is a trial use module for LSB (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it is the intention to make ALSA mandatory with the LSB 4.1 release targeted for Q1 2010. One of the requirements of a mandatory module is the existence of a test suite for the module that covers a good chunk of the interfaces provided by the LSB spec. An initial exploratory look at the existing test cases showed that tests appear to be tied to hardware drivers and are interactive.
Is this initial impression correct?
Yes.
From an LSB testing perspective, interface testing is very important; i.e. I call the interface with the arguments required and I get the expected result back.
In the case of a hardware interface library like libasound, there is not always a result that comes back where it could by easily tested.
While many parts of the library can be tested without special hardware (config, sequencer+MIDI, timers), the parts that are most likely to be used (PCM, mixer) require at least some kernel driver and contain many functions that are used to handle hardware differences.
1.) Is there interest from the community side to participate in this effort and accept patches?
I'm just an individual and cannot speak for "the community", whatever that is, but there is certainly interest to accept patches.
2.) Is it reasonable to expect that the existing tests can be used in some way by using a dummy sound device or a sound loop device to verify output?
There is a loopback driver, but it is not part of the LSB specification. This means that a test using this driver would not work with any alternate ALSA implementation.
I can easily imagine LSB-compliant computers that do not have any sound card (e.g., most servers). Even if the ALSA interface (i.e., libasound) is installed, some parts cannot be used in any meaningful way. What is a test supposed to do in this case? It could just exit with "pass", but this wouldn't actually _test_ much.
(Hmmm, a dummy ALSA implementation that just returns -ENODEV when the application tries to open a device would be compliant, because it cannot be distinguished from the 'real' ALSA with no installed sound card. :-)
3.) Would the community be more comfortable if this is an effort that creates new tests separate from the existing HW focused tests?
The current interactive tests are mostly intended to test drivers, not the interface. I don't think it would be desirable or even possible to integrate them into a test suite with strict pass/fail requirements.
4.) Is anyone interested in helping with this effort, writing tests, answering questions about ALSA for those not familiar with the interface and or sound in general?
Yes.
Since most of libasound is somewhat undocumented (the doxygen reference is just a skeleton), I'm wondering who is going to write the tests ...
Best regards, Clemens
At Tue, 16 Jun 2009 13:59:45 +0200, Clemens Ladisch wrote:
Robert Schweikert wrote:
Currently ALSA is a trial use module for LSB (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it is the intention to make ALSA mandatory with the LSB 4.1 release targeted for Q1 2010. One of the requirements of a mandatory module is the existence of a test suite for the module that covers a good chunk of the interfaces provided by the LSB spec. An initial exploratory look at the existing test cases showed that tests appear to be tied to hardware drivers and are interactive.
Is this initial impression correct?
Yes.
From an LSB testing perspective, interface testing is very important; i.e. I call the interface with the arguments required and I get the expected result back.
In the case of a hardware interface library like libasound, there is not always a result that comes back where it could by easily tested.
While many parts of the library can be tested without special hardware (config, sequencer+MIDI, timers), the parts that are most likely to be used (PCM, mixer) require at least some kernel driver and contain many functions that are used to handle hardware differences.
True. That's the very high hurdle that an automatic testing would face.
1.) Is there interest from the community side to participate in this effort and accept patches?
I'm just an individual and cannot speak for "the community", whatever that is, but there is certainly interest to accept patches.
Good to hear :)
2.) Is it reasonable to expect that the existing tests can be used in some way by using a dummy sound device or a sound loop device to verify output?
There is a loopback driver, but it is not part of the LSB specification. This means that a test using this driver would not work with any alternate ALSA implementation.
Yes, I mentioned about this in the last conf call with LSB guys.
Another candidate is alsa-lib file plugin (and null plugin). They can be used to simulate in a certain level of I/O.
I can easily imagine LSB-compliant computers that do not have any sound card (e.g., most servers). Even if the ALSA interface (i.e., libasound) is installed, some parts cannot be used in any meaningful way. What is a test supposed to do in this case? It could just exit with "pass", but this wouldn't actually _test_ much.
The test can never cover 100% all real cases. So, the primary question is: what kind of test do we need.
From the LSB perspective, I suppose, the API compatibility is the biggest issue. The functionality, either working or not on the real hardware, is a subsystem bug. And, the subtle tests would be likely needed done manually with the real hardware.
(Hmmm, a dummy ALSA implementation that just returns -ENODEV when the application tries to open a device would be compliant, because it cannot be distinguished from the 'real' ALSA with no installed sound card. :-)
You mean the dummy sound driver?
3.) Would the community be more comfortable if this is an effort that creates new tests separate from the existing HW focused tests?
The current interactive tests are mostly intended to test drivers, not the interface. I don't think it would be desirable or even possible to integrate them into a test suite with strict pass/fail requirements.
An automatic test would be hard, yes. But, including "manual tests" by human makes a lot sense. For example, providing a set of WAV files with different format, or giving a script to handle with sox, etc.
In my case, I've been testing like the following:
- Mixer adjustment via alsamixer - Play with different format bits (8, 16, 24, 32, float), sample rates, and channels via aplay - Play with different access methods (mmap, r/w) via aplay - Speaker configuration tests via speaker-test - Buffer size, period size tests via aplay - Recording via arecord; this is really hard to automate right now... - SPDIF AC3 output via ac3dec to a surround receiver - SPDIF input recording by cabling to SPDIF input
So, most of them are real hardware tests...
4.) Is anyone interested in helping with this effort, writing tests, answering questions about ALSA for those not familiar with the interface and or sound in general?
Yes.
Since most of libasound is somewhat undocumented (the doxygen reference is just a skeleton), I'm wondering who is going to write the tests ...
Heh, first response, first victim? :)
As mentioned, we need to focus on the definition of what kind of tests at first.
thanks,
Takashi
Takashi Iwai wrote:
Clemens Ladisch wrote:
There is a loopback driver, but it is not part of the LSB specification. This means that a test using this driver would not work with any alternate ALSA implementation.
Yes, I mentioned about this in the last conf call with LSB guys.
Another candidate is alsa-lib file plugin (and null plugin). They can be used to simulate in a certain level of I/O.
I don't think that LSB guarantees the availability of these plugins either. (And they don't get the timing correct, but maybe we don't care about that.)
I can easily imagine LSB-compliant computers that do not have any sound card (e.g., most servers). Even if the ALSA interface (i.e., libasound) is installed, some parts cannot be used in any meaningful way. What is a test supposed to do in this case? It could just exit with "pass", but this wouldn't actually _test_ much.
The test can never cover 100% all real cases. So, the primary question is: what kind of test do we need.
From the LSB perspective, I suppose, the API compatibility is the biggest issue. The functionality, either working or not on the real hardware, is a subsystem bug. And, the subtle tests would be likely needed done manually with the real hardware.
So ALSA's LSB tests would be like the Qt4 tests, whose description is: | check run-time presence of interfaces and absence of critical errors | in simple use cases
(Hmmm, a dummy ALSA implementation that just returns -ENODEV when the application tries to open a device would be compliant, because it cannot be distinguished from the 'real' ALSA with no installed sound card. :-)
You mean the dummy sound driver?
No. I meant that the ALSA interfaces, in case they are only required for strict LSB compliance but not for actual sound output, could be implemented like this to save memory:
int snd_pcm_open(...) { return -ENODEV; }
This is purely theoretical, but my point is that this would be a conforming implementation of the ALSA interface.
Best regards, Clemens
Clemens Ladisch wrote:
Takashi Iwai wrote:
Clemens Ladisch wrote:
There is a loopback driver, but it is not part of the LSB specification. This means that a test using this driver would not work with any alternate ALSA implementation.
Yes, I mentioned about this in the last conf call with LSB guys.
Another candidate is alsa-lib file plugin (and null plugin). They can be used to simulate in a certain level of I/O.
I don't think that LSB guarantees the availability of these plugins either. (And they don't get the timing correct, but maybe we don't care about that.)
If we need special libraries for testing that's fine. These do not have to be part of the LSB.
I can easily imagine LSB-compliant computers that do not have any sound card (e.g., most servers). Even if the ALSA interface (i.e., libasound) is installed, some parts cannot be used in any meaningful way. What is a test supposed to do in this case? It could just exit with "pass", but this wouldn't actually _test_ much.
The test can never cover 100% all real cases. So, the primary question is: what kind of test do we need.
From the LSB perspective, I suppose, the API compatibility is the biggest issue. The functionality, either working or not on the real hardware, is a subsystem bug. And, the subtle tests would be likely needed done manually with the real hardware.
So ALSA's LSB tests would be like the Qt4 tests, whose description is: | check run-time presence of interfaces and absence of critical errors | in simple use cases
Well, hopefully not quite that rudimentary but definitely in line with this train of thought. Just like the ALSA project cannot test all possible configurations, neither can the LSB. From a driver - hardware perspective the LSB probably can test less than those more deeply involved in the project.
One of the key parts of any effort is that the tests are accepted by the project and maintained in some way. If we just create LSB tests that are separate from the project things will diverge and become stale.
(Hmmm, a dummy ALSA implementation that just returns -ENODEV when the application tries to open a device would be compliant, because it cannot be distinguished from the 'real' ALSA with no installed sound card. :-)
You mean the dummy sound driver?
No. I meant that the ALSA interfaces, in case they are only required for strict LSB compliance but not for actual sound output, could be implemented like this to save memory:
int snd_pcm_open(...) { return -ENODEV; }
This is purely theoretical, but my point is that this would be a conforming implementation of the ALSA interface.
But this would fail the test where we load the dummy driver and test another return value (see previous post for more details).
Best regards, Clemens
On Wed, Jun 17, 2009 at 10:52:38AM +0200, Clemens Ladisch wrote:
Takashi Iwai wrote:
Yes, I mentioned about this in the last conf call with LSB guys.
Another candidate is alsa-lib file plugin (and null plugin). They can be used to simulate in a certain level of I/O.
I don't think that LSB guarantees the availability of these plugins either. (And they don't get the timing correct, but maybe we don't care about that.)
That's not necessarily fatal, *if* the LSB sound tests can include the plugin in question. Is the plugin interface a stable, published interface? That is, is it guaranteed to remain stable from release to release of ALSA? If so, then the test suite can provide the plugin, just for the purposes of running the test.
From the LSB perspective, I suppose, the API compatibility is the biggest issue. The functionality, either working or not on the real hardware, is a subsystem bug. And, the subtle tests would be likely needed done manually with the real hardware.
This is true, but some kind of minimal functionality testing is often useful to make sure that the *meaning* of the parameters haven't changed. For example, to take an extreme example, if a library were to accidentally change a long to "long long" in a function signature, but because of how other parameters and what happened to be in the registers on an x86 system, the program might not crash, and if were testing with a no-op output function for the program, we might not notice a problem.
So it's nice to be able to do a semantic test "does this function actually do roughly what it is spec'ed to do", and not just a simple API test. On the other hand, we also recognize that perfection can require far more resources than are available, so often we have to settle for a basic test.
- Ted
Theodore Tso wrote:
On Wed, Jun 17, 2009 at 10:52:38AM +0200, Clemens Ladisch wrote:
Takashi Iwai wrote:
Another candidate is alsa-lib file plugin (and null plugin). They can be used to simulate in a certain level of I/O.
I don't think that LSB guarantees the availability of these plugins either. (And they don't get the timing correct, but maybe we don't care about that.)
That's not necessarily fatal, *if* the LSB sound tests can include the plugin in question. Is the plugin interface a stable, published interface? That is, is it guaranteed to remain stable from release to release of ALSA?
Yes.
However, the plugin interface itself is not part of LSB. If there were some alternative ALSA implementation that wanted to be LSB conformant, the test would not work. (There are currently two alternatives, both named "SALSA", but neither one implements the full interface.)
Best regards, Clemens
Clemens Ladisch wrote:
Robert Schweikert wrote:
Currently ALSA is a trial use module for LSB (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it is the intention to make ALSA mandatory with the LSB 4.1 release targeted for Q1 2010. One of the requirements of a mandatory module is the existence of a test suite for the module that covers a good chunk of the interfaces provided by the LSB spec. An initial exploratory look at the existing test cases showed that tests appear to be tied to hardware drivers and are interactive.
Is this initial impression correct?
Yes.
From an LSB testing perspective, interface testing is very important; i.e. I call the interface with the arguments required and I get the expected result back.
In the case of a hardware interface library like libasound, there is not always a result that comes back where it could by easily tested.
While many parts of the library can be tested without special hardware (config, sequencer+MIDI, timers), the parts that are most likely to be used (PCM, mixer) require at least some kernel driver and contain many functions that are used to handle hardware differences.
Understood, but this is functional testing (current tests seem to cover this) and would required the appropriate hardware (a gazilion sound cards). LSB makes no guarantees that the drivers exist and have no bugs. LSB does strive to guarantee that a particular interface exists and behaves as specified.
So for interfaces that require a driver one could test that an appropriate error condition is returned if no driver exists, and we could test that no error condition is triggered when a dummy driver is connected. Correct?
1.) Is there interest from the community side to participate in this effort and accept patches?
I'm just an individual and cannot speak for "the community", whatever that is, but there is certainly interest to accept patches.
2.) Is it reasonable to expect that the existing tests can be used in some way by using a dummy sound device or a sound loop device to verify output?
There is a loopback driver, but it is not part of the LSB specification. This means that a test using this driver would not work with any alternate ALSA implementation.
I can easily imagine LSB-compliant computers that do not have any sound card (e.g., most servers). Even if the ALSA interface (i.e., libasound) is installed, some parts cannot be used in any meaningful way. What is a test supposed to do in this case? It could just exit with "pass", but this wouldn't actually _test_ much.
(Hmmm, a dummy ALSA implementation that just returns -ENODEV when the application tries to open a device would be compliant, because it cannot be distinguished from the 'real' ALSA with no installed sound card. :-)
Sorry, didn't quite get this, why a dummy ALSA implementation. Shouldn't the ALSA interfaces that depend on a driver already return -ENODEV today?
But you raise a good point w.r.t. hardware tie in. Within the LSB work group we are working on a concept to allow an application to test for module/component/subcomponent (name not yet decided) presence. For example a distribution can be LSB certified, but on a server install libasound may not be present. An LSB certified application at install time can query whether the LSB module/component/subcomponent that contains libasound is installed on the system and then take appropriate action. This would solve the problem you point out.
3.) Would the community be more comfortable if this is an effort that creates new tests separate from the existing HW focused tests?
The current interactive tests are mostly intended to test drivers, not the interface. I don't think it would be desirable or even possible to integrate them into a test suite with strict pass/fail requirements.
Good to know, sounds like we are better of starting from scratch.
4.) Is anyone interested in helping with this effort, writing tests, answering questions about ALSA for those not familiar with the interface and or sound in general?
Yes.
Since most of libasound is somewhat undocumented (the doxygen reference is just a skeleton), I'm wondering who is going to write the tests ...
I have some time to work on this, but not having much of a clue how sound works, and maybe that is not needed, I will need some amount of hand holding.
Robert
Best regards, Clemens
Robert Schweikert wrote:
Clemens Ladisch wrote:
While many parts of the library can be tested without special hardware (config, sequencer+MIDI, timers), the parts that are most likely to be used (PCM, mixer) require at least some kernel driver and contain many functions that are used to handle hardware differences.
Understood, but this is functional testing (current tests seem to cover this) and would required the appropriate hardware (a gazilion sound cards). LSB makes no guarantees that the drivers exist and have no bugs. LSB does strive to guarantee that a particular interface exists and behaves as specified.
So for interfaces that require a driver one could test that an appropriate error condition is returned if no driver exists, and we could test that no error condition is triggered when a dummy driver is connected. Correct?
Yes.
This would require that the test suite has the capability to load and unload kernel modules. Could this ever be a problem?
Best regards, Clemens
participants (4)
-
Clemens Ladisch -
Robert Schweikert -
Takashi Iwai -
Theodore Tso