Re: [alsa-devel] USB transfer_buffer allocations on 64bit systems
On Tue, 11 May 2010, FUJITA Tomonori wrote:
At one point we tried an experiment, printing out the buffer and DMA addresses. I don't recall seeing anything obviously wrong, but if an IOMMU was in use then that might not mean anything. Is it possible that the IOMMU mappings sometimes get messed up for addresses above 4 GB?
You mean that an IOMMU could allocate an address above 4GB wrongly? If so, IIRC, all the IOMMU implementations use dev->dma_mask and dev->coherent_dma_mask properly. And the DMA address space of the majority of IOMMUs are limited less than 4GB.
The Intel IOMMU code will use dev->dma_mask and dev->coherent_dma_mask properly. It is not limited to 4GiB, but it will tend to give virtual DMA addresses below 4GiB even when a device is capable of more; it'll only give out higher addresses when the address space below 4GiB is exhausted.
What I meant was: Is it possible that the IOMMU code will return a virtual DMA address before 4 GB but will somehow forget to actually map that address to the data buffer?
Then, the IOMMU is completely broken. Then, we would get tons of DMA bugs not only about USB, I guess. So I'm not sure.
Yes, you're right about that.
The problem goes away when Pedro boots with mem=4G. And the dma_mask value is set properly (in fact, the ehci-hcd driver currently doesn't use 64-bit DMA at all).
If anyone wants to see the debug log entries showing the buffer and DMA addresses, they are attached to this email message:
http://marc.info/?l=linux-kernel&m=127076841801054&w=2
Either the data isn't getting written to the buffer correctly or else the buffer isn't getting sent to the device correctly. Can anybody suggest a means of determining which is the case?
I can't say anything about this log that including only DMA addresses. I'm not familiar with how the USB core does DMA stuff. And the USB stack design that the USB core does DMA stuff (allocating, mappings, etc) makes debugging DMA issues really difficult.
The DMA stuff is simple enough in this case. The urb->transfer_buffer address is passed to dma_map_single(), and the DMA address it returns is stored in urb->transfer_dma. Those are the two values printed out by the debugging patch.
Alan Stern
On Tue, 11 May 2010 10:00:50 -0400 (EDT) Alan Stern stern@rowland.harvard.edu wrote:
The problem goes away when Pedro boots with mem=4G. And the dma_mask value is set properly (in fact, the ehci-hcd driver currently doesn't use 64-bit DMA at all).
If anyone wants to see the debug log entries showing the buffer and DMA addresses, they are attached to this email message:
http://marc.info/?l=linux-kernel&m=127076841801054&w=2
Either the data isn't getting written to the buffer correctly or else the buffer isn't getting sent to the device correctly. Can anybody suggest a means of determining which is the case?
I can't say anything about this log that including only DMA addresses. I'm not familiar with how the USB core does DMA stuff. And the USB stack design that the USB core does DMA stuff (allocating, mappings, etc) makes debugging DMA issues really difficult.
The DMA stuff is simple enough in this case. The urb->transfer_buffer address is passed to dma_map_single(), and the DMA address it returns is stored in urb->transfer_dma. Those are the two values printed out by the debugging patch.
You've already confirmed that all the returned addresses (urb->transfer_dma) in the log is less than 4GB, right?
Drivers need to take care of more things (not only about addresses) to do DMA properly, that is, use the DMA API in the proper way (such as, once calling dma_map_single, drivers can't touch the buffer until calling dma_unmap_single)
Using the DMA API and performing DMA transfer in different places just make things complicated.
Either the data isn't getting written to the buffer correctly or else the buffer isn't getting sent to the device correctly. Can anybody suggest a means of determining which is the case?
I can't say anything about this log that including only DMA addresses. I'm not familiar with how the USB core does DMA stuff. And the USB stack design that the USB core does DMA stuff (allocating, mappings, etc) makes debugging DMA issues really difficult.
The DMA stuff is simple enough in this case. The urb->transfer_buffer address is passed to dma_map_single(), and the DMA address it returns is stored in urb->transfer_dma. Those are the two values printed out by the debugging patch.
Is that address (urb->transfer_dma) the same as 'virt_to_phys(urb->transfer_buffer)' (if not, then SWIOTLB is being utilized) and is the dma_sync_* done on the urb->transfer_dma (to properly sync the data from the SWIOTLB to the transfer_buffer) before you start using the urb->transfer_buffer?
On Tue, 11 May 2010 10:24:40 -0400 Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Either the data isn't getting written to the buffer correctly or else the buffer isn't getting sent to the device correctly. Can anybody suggest a means of determining which is the case?
I can't say anything about this log that including only DMA addresses. I'm not familiar with how the USB core does DMA stuff. And the USB stack design that the USB core does DMA stuff (allocating, mappings, etc) makes debugging DMA issues really difficult.
The DMA stuff is simple enough in this case. The urb->transfer_buffer address is passed to dma_map_single(), and the DMA address it returns is stored in urb->transfer_dma. Those are the two values printed out by the debugging patch.
Is that address (urb->transfer_dma) the same as 'virt_to_phys(urb->transfer_buffer)' (if not, then SWIOTLB is being utilized) and is the dma_sync_* done on the urb->transfer_dma (to properly sync the data from the SWIOTLB to the transfer_buffer) before you start using the urb->transfer_buffer?
Or calling dma_unmap_single.
Can you tell me all the exact process of DMA that the usb core and the driver do?
On Tue, 11 May 2010, FUJITA Tomonori wrote:
On Tue, 11 May 2010 10:24:40 -0400 Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Either the data isn't getting written to the buffer correctly or else the buffer isn't getting sent to the device correctly. Can anybody suggest a means of determining which is the case?
I can't say anything about this log that including only DMA addresses. I'm not familiar with how the USB core does DMA stuff. And the USB stack design that the USB core does DMA stuff (allocating, mappings, etc) makes debugging DMA issues really difficult.
The DMA stuff is simple enough in this case. The urb->transfer_buffer address is passed to dma_map_single(), and the DMA address it returns is stored in urb->transfer_dma. Those are the two values printed out by the debugging patch.
Is that address (urb->transfer_dma) the same as 'virt_to_phys(urb->transfer_buffer)' (if not, then SWIOTLB is being utilized) and is the dma_sync_* done on the urb->transfer_dma (to properly sync the data from the SWIOTLB to the transfer_buffer) before you start using the urb->transfer_buffer?
Or calling dma_unmap_single.
Can you tell me all the exact process of DMA that the usb core and the driver do?
1. The audio driver stores data in urb->transfer_buffer.
2. The audio driver calls usb_submit_urb(urb).
3. The USB core does urb->transfer_dma = dma_map_single(controller, urb->transfer_buffer, urb->transfer_buffer_length, DMA_TO_DEVICE);
4. The host controller driver tells the hardware to carry out the data transfer.
5. When the hardware says the transfer is finished, the USB core does dma_unmap_single(controller, urb->transfer_dma, urb->transfer_buffer_length, DMA_TO_DEVICE);
6. The audio driver's completion handler is called: (urb->complete)(urb);
Alan Stern
On Tue, 11 May 2010 11:04:55 -0400 (EDT) Alan Stern stern@rowland.harvard.edu wrote:
On Tue, 11 May 2010, FUJITA Tomonori wrote:
On Tue, 11 May 2010 10:24:40 -0400 Konrad Rzeszutek Wilk konrad.wilk@oracle.com wrote:
Either the data isn't getting written to the buffer correctly or else the buffer isn't getting sent to the device correctly. Can anybody suggest a means of determining which is the case?
I can't say anything about this log that including only DMA addresses. I'm not familiar with how the USB core does DMA stuff. And the USB stack design that the USB core does DMA stuff (allocating, mappings, etc) makes debugging DMA issues really difficult.
The DMA stuff is simple enough in this case. The urb->transfer_buffer address is passed to dma_map_single(), and the DMA address it returns is stored in urb->transfer_dma. Those are the two values printed out by the debugging patch.
Is that address (urb->transfer_dma) the same as 'virt_to_phys(urb->transfer_buffer)' (if not, then SWIOTLB is being utilized) and is the dma_sync_* done on the urb->transfer_dma (to properly sync the data from the SWIOTLB to the transfer_buffer) before you start using the urb->transfer_buffer?
Or calling dma_unmap_single.
Can you tell me all the exact process of DMA that the usb core and the driver do?
- The audio driver stores data in urb->transfer_buffer.
How urb->transfer_buffer is allocated?
The audio driver calls usb_submit_urb(urb).
The USB core does urb->transfer_dma = dma_map_single(controller, urb->transfer_buffer, urb->transfer_buffer_length, DMA_TO_DEVICE);
The mapping error is checked via dma_mapping_error? Even if an mapping error happens, no data corruption happens due to that (needs something like retrying the request)?
The host controller driver tells the hardware to carry out the data transfer.
When the hardware says the transfer is finished, the USB core does dma_unmap_single(controller, urb->transfer_dma, urb->transfer_buffer_length, DMA_TO_DEVICE);
The audio driver's completion handler is called: (urb->complete)(urb);
The driver does only DMA_TO_DEVICE? Or you see DMA problems only with DMA_TO_DEVICE?
participants (3)
-
Alan Stern -
FUJITA Tomonori -
Konrad Rzeszutek Wilk