[alsa-devel] Allocating buffers for USB transfers (again)
I have to revisit an issue we've discussed in length around a year ago and which still remains unsolved. I've been getting feedback from more users of my driver snd-usb-caiaq who report the issue found in bug #15580 in the kernel bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=15580
Let me quickly summarize the current state of this issue as I see it.
The problem seems to be that certain 64bit chipsets can't deal with the fact that an URB's transfer_buffer is allocated with kmalloc(GFP_KERNEL). The effect is that kmalloc() is very likely to hand out memory which is not addressable by devices that are connected via 32bit PCI busses, such as EHCI controllers. In theory, DMA bounce buffers should be installed in such cases, or the IOMMU would be in charge to re-map these buffers to suitable locations, but for at least two people who have reported the issue, this obviously fails.
So my approach was to force the driver using memory that is DMA coherent, which leads to 32bit addressable memory if the device's DMA_MASK is set accordingly. Other drivers do the same, and the patch for doing this is found here: http://lkml.org/lkml/2010/5/7/61. Users that see the bug all report that this patch fixes their problem with my driver. However, it was rejected back then as other developers said it was fixing the wrong end, and that the driver doesn't necessarily need coherent memory.
Takashi recently posted a patch to the bugzilla entry which uses a different approach: it introduces a function to determine suitable GFP flags for USB devices, and passes __DMA32 to kmalloc() eventually. However, using this flags directly with the SLUB allocator is illegal and causes a BUG() in mm/slub.c, cache_grow().
The question now is how to proceed. I see three possible ways.
1. Find a way to allocate 32bit-aware memory with kmalloc(), following Takashi's idea 2. Find out exactly why these machines fail to install bounce buffers or set up their IOMMU correctly 3. re-activate the currently disabled functions usb_buffer_{map,unmap,sync} functions and let the USB stack do the memory mapping
I'm not really deep into all the memory management code and even less into how the IOMMU and bounce buffers work, so I need some help here. Can anyone elaborate what would be the best option? I really want to get this fixed now, and provide a driver that works for everyone.
What really puzzles me is that this doesn't hit a lot more people with other drivers.
Thanks, Daniel
Daniel Mack wrote:
I have to revisit an issue we've discussed in length around a year ago and which still remains unsolved. I've been getting feedback from more users of my driver snd-usb-caiaq who report the issue found in bug #15580 in the kernel bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=15580
Let me quickly summarize the current state of this issue as I see it.
The problem seems to be that certain 64bit chipsets can't deal with the fact that an URB's transfer_buffer is allocated with kmalloc(GFP_KERNEL). The effect is that kmalloc() is very likely to hand out memory which is not addressable by devices that are connected via 32bit PCI busses, such as EHCI controllers. In theory, DMA bounce buffers should be installed in such cases, or the IOMMU would be in charge to re-map these buffers to suitable locations, but for at least two people who have reported the issue, this obviously fails.
The problem is that snd-usb-caiaq modifies the URB buffer after submission.
USB drivers must always be prepared to work with host controllers that use double buffering. (IIRC there are some exotic ones where the hardware works only with internal SRAM.) DMA mapping, if available, is only an optimization.
The question now is how to proceed. I see three possible ways. ... 3. re-activate the currently disabled functions usb_buffer_{map,unmap,sync} functions and let the USB stack do the memory mapping
This is the only way that allows bounce buffers to be copied at the correct time.
What really puzzles me is that this doesn't hit a lot more people with other drivers.
I don't think there is any other driver that relies on the buffer being coherently DMA-mapped.
Regards, Clemens
On Thu, Jul 7, 2011 at 2:14 PM, Clemens Ladisch clemens@ladisch.de wrote:
Daniel Mack wrote:
I have to revisit an issue we've discussed in length around a year ago and which still remains unsolved. I've been getting feedback from more users of my driver snd-usb-caiaq who report the issue found in bug #15580 in the kernel bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=15580
Let me quickly summarize the current state of this issue as I see it.
The problem seems to be that certain 64bit chipsets can't deal with the fact that an URB's transfer_buffer is allocated with kmalloc(GFP_KERNEL). The effect is that kmalloc() is very likely to hand out memory which is not addressable by devices that are connected via 32bit PCI busses, such as EHCI controllers. In theory, DMA bounce buffers should be installed in such cases, or the IOMMU would be in charge to re-map these buffers to suitable locations, but for at least two people who have reported the issue, this obviously fails.
The problem is that snd-usb-caiaq modifies the URB buffer after submission.
Hmm, no, it doesn't. I know I stated that once, but I confused the implementation of the Linux driver with the approach I chose for the Mac OS X driver code. Sorry about that.
The driver simply copies audio material to its output URB and calls usb_submit_urb(). The URB buffer is not touched after that call.
Daniel
On Thu, Jul 07, 2011 at 01:53:46PM +0200, Daniel Mack wrote:
So my approach was to force the driver using memory that is DMA coherent, which leads to 32bit addressable memory if the device's DMA_MASK is set accordingly. Other drivers do the same, and the patch for doing this is found here: http://lkml.org/lkml/2010/5/7/61. Users that see the bug all report that this patch fixes their problem with my driver. However, it was rejected back then as other developers said it was fixing the wrong end, and that the driver doesn't necessarily need coherent memory.
I think this is a valid approach, and since it avoids using bounce buffers it is potentially better wrt performance. BTW, usb_alloc_coherent() doc comment says you need to set the URB_NO_TRANSFER_DMA_MAP flag, which your patch fails to do.
Takashi recently posted a patch to the bugzilla entry which uses a different approach: it introduces a function to determine suitable GFP flags for USB devices, and passes __DMA32 to kmalloc() eventually. However, using this flags directly with the SLUB allocator is illegal and causes a BUG() in mm/slub.c, cache_grow().
from gfp.h: /* Do not use these with a slab allocator */ #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
I think it means you can only use __GFP_DMA32 for get_free_pages(), not for kmalloc().
- Find out exactly why these machines fail to install bounce buffers
or set up their IOMMU correctly
I think this needs to be done.
Johannes
participants (3)
-
Clemens Ladisch -
Daniel Mack -
Johannes Stezenbach