About a year ago I posted an explaination of the cause of non-functioning with Alsa-OSS in duplex, together with a workaround patch. After some discussion the thread ended. However since then there has been a steady trickle of downloads of the patches from my host (it can also be found elsewhere) and some reports of great success,eg
http://audacityteam.org/forum/thread/1388 (2nd page especially)
In attempting to tidy up some loose ends, I hope you can bear with me revisiting this and making the case for it's acceptance. I am aware the Alsa-OSS is not of much interest to developers, and it might be said 'use native Alsa'; however that is not possible or desirable in all applications, and anyway, why provide an OSS emulation that cannot be made to work properly?
To summarise the problem first:
OSS provides a single SNDCTL_DSP_CHANNELS ioctl; where a device is duplex the number of capture and playback channels cannot be set separately. With original OSS this was not a problem AFAIK because separate devices are created for capture and playback, ie they are not used duplex. However Alsa-OSS in most cases creates a single combined device, which only operates correctly when an equal number of capture and playback channels are used.
The Workaround:
The workaround I proposed and now use constantly is like a secret trapdoor. Where the existing interface is adequate (ie captue channels == playback channels, or non-duplex) nothing changes, and SNDCTL_DSP_CHANNELS is set with a number between 0 and 128. It is fully compatible with apps that don't know of the secret door -the front door is used!
However, if an app needs to set capture channels != playback channels, in duplex, at present the stream will malfunction every time. But with this patch applied, if the app knows the workaround, then it can simply set SNDCTL_DSP_CHANNELS as playback + 256*capture channels, and this will be interpreted correctly in the patched Alsa-OSS and the stream functions correctly.
As I say, there is therefore no loss of compatibility in either direction. Providing the app only uses the special formula when necessary rather than always, it is only providing a way to fix something that is broken. It would seem to me of value to add the patch to Alsa so that the workaround can be effected by changes to the app only, without also having to patch and compile a replacement Alsa. Whilst it is by no means ideal, it does overcome the underlying problem successfully. Details of the workaround could be added to the OSS-Emulation.html document.
Patch for 1.0.14rc4 is attached for consideration, together with example portaudio patch.
Regards Alan